Distributed inference among multiple CPU nodes with TCP failed

[adamcchen]

Describe the issue

When I trying to run distributed inference among multiple CPU nodes with TCP, it's blocked with the output below,

Loading 34 checkpoint shards: 100%|██████████| 34/34 [00:10<00:00,  3.18it/s]
Loading 34 checkpoint shards: 100%|██████████| 34/34 [00:10<00:00,  3.15it/s]
Loading 34 checkpoint shards: 100%|██████████| 34/34 [00:10<00:00,  3.39it/s][2025-04-29 03:59:28,612] [INFO] [utils.py:781:see_memory_usage] post-ds-inference-init
[2025-04-29 03:59:28,612] [INFO] [utils.py:782:see_memory_usage] MA 14.31 GB         Max_MA 14.31 GB         CA 14.31 GB         Max_CA 14 GB
[2025-04-29 03:59:28,612] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 35.0 GB, percent = 13.9%
Loading 34 checkpoint shards: 100%|██████████| 34/34 [00:10<00:00,  3.10it/s]
Loading 34 checkpoint shards: 100%|██████████| 34/34 [00:11<00:00,  3.04it/s]
2025-04-29 03:59:28,960 - run_generation_with_deepspeed.py - __main__ - DEBUG - Applying ipex.llm.optimize on rank 0/4
2025-04-29 03:59:29,034 - run_generation_with_deepspeed.py - __main__ - DEBUG - Applying ipex.llm.optimize on rank 1/4
2025-04-29 03:59:28,960 - run_generation_with_deepspeed.py - __main__ - DEBUG - Applying ipex.llm.optimize on rank 1/4
2025-04-29 03:59:29,035 - run_generation_with_deepspeed.py - __main__ - DEBUG - Applying ipex.llm.optimize on rank 0/4
2025-04-29 03:59:28,962 - optimize.py - IPEX - DEBUG - ipex.llm.optimize is converting model to reference model
2025-04-29 03:59:28,962 - optimize.py - IPEX - DEBUG - ipex.llm.optimize is converting model to reference model
2025-04-29 03:59:29,036 - optimize.py - IPEX - DEBUG - ipex.llm.optimize is converting model to reference model
2025-04-29 03:59:29,036 - optimize.py - IPEX - DEBUG - ipex.llm.optimize is converting model to reference model
Bad file descriptor.: No such file or directory
Bad file descriptor.: No such file or directory
2025-04-29 03:59:32,984 - optimize.py - IPEX - DEBUG - ipex.llm.optimize is lowering model
2025-04-29 03:59:32,996 - _logger.py - IPEX - INFO - Conv BatchNorm folding failed during the optimize process.
2025-04-29 03:59:33,000 - optimize.py - IPEX - DEBUG - ipex.llm.optimize is lowering model
2025-04-29 03:59:33,003 - _logger.py - IPEX - INFO - Linear BatchNorm folding failed during the optimize process.
2025-04-29 03:59:33,011 - _logger.py - IPEX - INFO - Conv BatchNorm folding failed during the optimize process.
2025-04-29 03:59:33,018 - _logger.py - IPEX - INFO - Linear BatchNorm folding failed during the optimize process.
2025-04-29 03:59:34,837 - run_generation_with_deepspeed.py - __main__ - DEBUG - Applying ipex.llm.optimize done on rank 0/4
2025-04-29 03:59:34,861 - run_generation_with_deepspeed.py - __main__ - DEBUG - Applying ipex.llm.optimize done on rank 1/4

Bad file descriptor.: No such file or directory should be the cause of the blocking issue. I have been working on it for couple of days, help pls~

The steps to reproduce this issue,

* # Build an image with the provided Dockerfile by installing from Intel® Extension for PyTorch\* prebuilt wheel files
# To have a custom ssh server port for multi-nodes run, please add --build-arg PORT_SSH=<CUSTOM_PORT> ex: 2345, otherwise use the default 22 SSH port
DOCKER_BUILDKIT=1 docker build -f examples/cpu/llm/Dockerfile \
  --build-arg COMPILE=ON \
  --build-arg PORT_SSH=2345 \
  -t ipex-llm:2.7.0 .

# Run the container with command below
sudo docker run --rm -it --privileged \
  -v /dev/shm:/dev/shm \
  --net host \
  -v /home/sietium/.cache/huggingface/hub/models--LLM-Research--Meta-Llama-3.1-8B-Instruct:/root/.cache/huggingface/hub/models--LLM-Research--Meta-Llama-3.1-8B-Instruct \
  ipex-llm:2.7.0 bash

# When the command prompt shows inside the docker container, enter llm examples directory
cd llm

# Activate environment variables
# set bash script argument to "inference" or "fine-tuning" for different usages
source ./tools/env_activate.sh inference

python utils/create_shard_model.py -m /root/.cache/huggingface/hub/models--LLM-Research--Meta-Llama-3.1-8B-Instruct --save-path ./local_llama3_1_8b

# update the configurations in tools/run_scaling.sh

bash -x tools/run_scaling.sh

The cpu of two hosts is,

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   64
  On-line CPU(s) list:    0-63
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz
    CPU family:           6
    Model:                106
    Thread(s) per core:   2
    Core(s) per socket:   16
    Socket(s):            2
    Stepping:             6
    CPU max MHz:          3400.0000
    CPU min MHz:          800.0000
    BogoMIPS:             4800.00
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic mov
                          be popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd s
                          ha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabi
                          lities

[louie-tsai]

run_scaling.sh doesn't seem to support shard model
could you follow the instructions in below section?
https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm/inference#242-distributed-inference-with-xeon-cpu-max-series

[adamcchen]

run_scaling.sh doesn't seem to support shard model could you follow the instructions in below section? https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm/inference#242-distributed-inference-with-xeon-cpu-max-series

I did shard model before running run_scaling.sh just as your link said. And your link only provide the instructions for single host which contain one or multiple CPUs, my case is multiple hosts.

I have tried the sharded model and non-sharded model, all failed. It's right on single host case.

[adamcchen]

There are 2 hosts, with 2 CPUs on each host, with 16 cores on each CPU.

The two hosts are in the same local network, and running the same docker image provided by Intel.

The CPU is the third-generation xeon intel CPUs which support avx512 and avx512-vnni.

When only enable one host in hostfile.txt, everything works well.
When enable all the two hosts in hostfile.txt, it stucks and output a error message "Bad file descriptor.: No such file or directory"
When enable all the two hosts in hostfile.txt, and remove --ipex in run_scaling.sh, it works well.

So does ipex.llm.optimize support distribute inference on multi-host case?

@ZailiWang @blzheng @jianan-gu

[ZailiWang]

Hi @adamcchen , would you paste the full error call stack? Thanks!
Oh, I see that the the top. We'll check.

[ZailiWang]

Hi, generally speaking, multi-node usage is not a prioritized scenario for CPU workloads. May I know what is your scenario/background to run multi-node?

[adamcchen]

Hi, generally speaking, multi-node usage is not a prioritized scenario for CPU workloads. May I know what is your scenario/background to run multi-node?

When running with only one host, it can't reach the 10 tokens/s. We want try multi-node to achieve it.

[ZailiWang]

I checked with the internal developer team but unfortunately this script is deprecated.
When the script was created it was expected to bring in further performance boost (as you tried), but later we found that in LLM decoding phase, the data synchronization speed (rather than CPU core computing speed) is the bottleneck, and the cross-node data transferring with TCP was even much slower than the shared memory solution for intra-node data transferring. Hence, the overall performance would be even worse.

That's why the script was not maintained for a while and it's not compatible with latest IPEX core codes now. We will totally remove this script in the next release. Sorry for the inconvenience and confusion for you.

[adamcchen]

Thanks. We have tried another solution, xFasterTransformer, which support multi-node and it can reach 10 tokens/s with six devices.