apptainer docker tensorflow container issue with 2.18 - skipping loading of GPU

[ashep29]

I'm using apptainer pull docker://tensorflow/tensorflow:latest-gpu, but tensorflow 2.18 is skipping loading of GPU, with this message:

W0000 00:00:1736383795.205652 747392 gpu_device.cc:2344] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at [https://www.tensorflow.org/install/gpu] for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

The tensor 2.17 container worked fine, but it appears that the 2.18 container wants to use libcudnn.9.0 but the container only provides libcudnn.8.0. The system libcudnn is not mapped to the container. The system has both 8 and 9 installed. This looks like a bug in the build of the container.

Note: Just using the system python and installing 2.18 (no containers), works fine as the system has both cudnn 8 and 9 installed.

[ngaywood]

The tensorflow 2.18 tried to open libcudnn.so.9 that is missing from the docker container.

This has also been reported here [libcudnn.so.9 missing] (tensorflow/tensorflow#80538 (comment))

[joshcaughtfire]

Afaik, based on this https://www.tensorflow.org/install/source#gpu it appears that 2.18 requires cuDNN 9.3, however the current build (https://github.com/tensorflow/build/blob/master/tensorflow_runtime_dockerfiles/gpu.packages.txt) is installing 8.9.6.50. From my understanding, that's the version last used with 2.13 and actually might explain some of the stability issues I've seen training on the docker image for recent versions.

Building an docker image from the official image with that lib installs seems to resolve it:

FROM tensorflow/tensorflow:2.18.0-gpu

RUN apt update && apt install -y --no-install-recommends libcudnn9-cuda-12=9.3.0.75-1

Fixing the official image would be preferred, but this has worked on my project. I haven't looked at how it is built enough to know if it's as simple as updating the gpu.packages.txt or if it needs to be updated based on the version it's being built against as I think this is built internally at Google and not sure their setup.

[json-han]

I was surprised to find that this issue still hasn't been fixed.

I tried to use the latest TensorFlow by running the tensorflow/tensorflow:latest-gpu-jupyter image, which currently ships with TensorFlow 2.19.0.

After launching a container from this image and checking cuDNN with dpkg -l | grep cudnn, I found the following result:

ii  libcudnn8  8.9.6.50-1+cuda12.2  amd64  cuDNN runtime libraries

This means that the official image includes cuDNN 8.9, even though TensorFlow 2.19.0 requires cuDNN 9.3 or higher. (https://www.tensorflow.org/install/source#gpu)

To resolve this issue inside the container, I followed these steps:

1. Remove existing cuDNN

   apt-get remove --purge libcudnn*

2. Install wget

   apt install wget

3. Download and install the latest cuDNN from the official NVIDIA repository

   wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
   dpkg -i cuda-keyring_1.1-1_all.deb
   apt-get update
   apt-get -y install cudnn

Note: Since all commands inside the Docker container are run as root, sudo is not required.

After step 3, cuDNN 9.10 was successfully installed in my container.

I later found that some others resolved this issue using the following commands:

apt update \
    && apt install -y --no-install-recommends libcudnn9-cuda-12 \
    && rm -rf /var/lib/apt/lists/*

This workaround also installs the required cuDNN 9.x version.