DDP NCCL freezes in docker AWS Jupyter

[awaelchli]

❓ Questions and Help

Problem

Jupyter terminal freezes, and connection to AWS node closes.
The problem is reproducible with any Lightning example.

What have you tried?

python pl_examples/basic_examples/image_classifier.py --gpus 4 --accelerator ddp

What's your environment?

Linux using docker image
Lightning 1.0.4
pytorch 1.6
cuda 10.2

[awaelchli]

The solution was to use

export NCCL_SOCKET_IFNAME=lo