[Bug]: vLLM failed to load model with TP >= 2 on Gaudi 3

[jerin-scalers-ai]

Your current environment

Docker image of built using below command:
docker build -f Dockerfile.hpu -t vllm-hpu-env .

🐛 Describe the bug

The vLLM failed to load model on multi gpu when we set the tensor parallelism greater than or equal to 2 on a 8x Gaudi 3 system.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ClayMetrumAI]

I experienced this same issue... Loading the Nvidia quantized Llama 3.3 70B with FP8 on 8xGaudi3 failed with the inability to load on multiple HPUS, using the same vLLM image. Here is a screenshot of the issue as well:

[jerin-scalers-ai]