"Could not load dynamic library 'libcudnn.so.8'" when running tensorflow on ubuntu 20.04

Question

Note: there are many similar questions but for different versions of ubuntu and somewhat different specific libraries. I have not been able to figure out what combination of symbolic links, additional environment variables such as LD_LIBRARY_PATH would work

Here is my nvidia configuration

$ nvidia-smi
Tue Apr  6 11:35:54 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| 18%   25C    P8     9W / 175W |     25MiB /  7982MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1081      G   /usr/lib/xorg/Xorg                 20MiB |
|    0   N/A  N/A      1465      G   /usr/bin/gnome-shell                3MiB |
+-----------------------------------------------------------------------------+

When running a TF program the following happened:

2021-04-06 14:35:01.589906: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2021-04-06 14:35:01.589914: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

Has anyone seen this particular mix and how did you resolve it?

Here is one of the additional fixes attempted, but with no change:

conda install cudatoolkit=11.0

Answer 1

So I had the same issue. As the comments say, it's because you need to install CUDNN. For that, there is a guide here.

But as I know already your distro (Ubuntu 20.04) I can give you the command lines already:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/${last_public_key}.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
sudo apt-get install libcudnn8
sudo apt-get install libcudnn8-dev

where ${last_public_key} is the last public key (file with .pub extension) published on https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/. (At May 9th 2022 when this post was edit, it was 3bf863cc.pub).

And if you want to install a specific version, the last 2 commands would be replaced with

sudo apt-get install libcudnn8=${cudnn_version}-1+${cuda_version}
sudo apt-get install libcudnn8-dev=${cudnn_version}-1+${cuda_version}

where ${cudnn_version} is for example 8.2.4.* and ${cuda_version} is for example cuda11.0 (as I see you have 11.0 on the command nvidia-smi, although I have not tested it as mine was 11.4 but I guess it should work Ok)

Answer 2

I had the same issue and linux OS is Centos7-6 for me. Since I don't have sudo access on this machine, I solved the issue by installing cudnn from the anaconda website

In the environment where the latest tensorflow flow is installed :

conda install -c anaconda cudnn

You can check the package install using conda list : ( I had previously installed the cudatoolkit from anaconda )

Name                      Version              Build  Channel
cudatoolkit               11.3.1               h2bc3f7f_2
cudnn                     8.2.1                cuda11.3_0

You can check if tensorflow and the gpus are talking to each other :

(tf2_6) [xxxx@msird-gpu-01 envs]$ python
Python 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> gpus = tf.config.experimental.list_physical_devices('GPU')
>>> for gpu in gpus:
...     print("Name:", gpu.name, "  Type:", gpu.device_type)
...
Name: /physical_device:GPU:0   Type: GPU
Name: /physical_device:GPU:1   Type: GPU

Answer 3

I used in Ubuntu 22.04

sudo apt install nvidia-cudnn