[Usage]: How to implement the inference test of LLM model PD (Prefill-Decode) disaggregation using the vllm framework ?

[Alan-D-Chen]

Your current environment

Dear vllm developers,

First of all, on behalf of our university laboratory, I would like to express our highest gratitude and respect for the work you have done. However, for students like us, it is quite challenging to implement the inference test of LLM model PD (Prefill-Decode) disaggregation using the vllm framework. Is there a manual or tutorial available to help beginners carry out the inference test of LLM model PD disaggregation? We offer our sincere thanks and admiration once again.

New we have installed vllm 0.9.2 in our servers.

root@5468a704:/workspace/vllm-0.9.2# pip show vllm
Name: vllm
Version: 0.9.2
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email:
License-Expression: Apache-2.0
Location: /usr/local/lib/python3.12/dist-packages
Requires: aiohttp, blake3, cachetools, cloudpickle, compressed-tensors, depyf, einops, fastapi, filelock, gguf, huggingface-hub, lark, llguidance, lm-format-enforcer, mistral_common, msgspec, ninja, numba, numpy, openai, opencv-python-headless, outlines, partial-json-parser, pillow, prometheus-fastapi-instrumentator, prometheus_client, protobuf, psutil, py-cpuinfo, pybase64, pydantic, python-json-logger, pyyaml, pyzmq, ray, regex, requests, scipy, sentencepiece, setuptools, six, tiktoken, tokenizers, torch, torchaudio, torchvision, tqdm, transformers, typing_extensions, watchfiles, xformers, xgrammar
Required-by:

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[wuhang2014]

Hi @Alan-D-Chen

Thank you very much for your kind words and for your interest in vLLM! We're happy to support your research and help you get started with PD (Prefill-Decode) disaggregation.

To run inference tests using PD disaggregation with vLLM, we recommend checking out these two example guides in our documentation:

• Disaggregated Prefill Example:
  This guide demonstrates how to run the prefill (initial token computation) phase separately using the vLLM online serving API.

• Disaggregated Serving Example:
  This example expands on the first, showing you how to perform both the prefill and decode phases as separate steps, enabling you to control and test each part of the inference process.

These guides include sample client code and API usage that should help you get started. Typically, the workflow involves:

1. Calling the prefill endpoint to process your prompts and obtain the necessary states.
2. Using the decode endpoint to generate tokens incrementally based on those states.

If you have already installed vLLM 0.9.2, you can adapt the examples to your environment. The guides offer Python code snippets that you can run as-is or modify for your specific model and use case.

If you have further questions or run into any issues while following these examples, please let us know! We're happy to help clarify or provide more detailed instructions as needed.

Thank you for your support and best of luck with your research!

[Alan-D-Chen]

Thank you very much for your enthusiastic help @wuhang2014 . We are greatly encouraged and will make immediate attempts. Here is our background information:

root@5468a704:/workspace/vllm-0.9.2# pip show vllm
Name: vllm
Version: 0.10.0
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email: 
License-Expression: Apache-2.0
Location: /usr/local/lib/python3.12/dist-packages
Requires: aiohttp, blake3, cachetools, cbor2, cloudpickle, compressed-tensors, depyf, diskcache, einops, fastapi, filelock, gguf, huggingface-hub, lark, llguidance, lm-format-enforcer, mistral_common, msgspec, ninja, numba, numpy, openai, opencv-python-headless, outlines_core, partial-json-parser, pillow, prometheus-fastapi-instrumentator, prometheus_client, protobuf, psutil, py-cpuinfo, pybase64, pydantic, python-json-logger, pyyaml, pyzmq, ray, regex, requests, scipy, sentencepiece, setuptools, six, tiktoken, tokenizers, torch, torchaudio, torchvision, tqdm, transformers, typing_extensions, watchfiles, xformers, xgrammar
Required-by: 

root@5468a704:/workspace/vllm-0.9.2# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Apr__9_19:24:57_PDT_2025
Cuda compilation tools, release 12.9, V12.9.41
Build cuda_12.9.r12.9/compiler.35813241_0
root@5468a704:/workspace/vllm-0.9.2# nccl-tests --version
bash: nccl-tests: command not found
root@5468a704:/workspace/vllm-0.9.2# find / -name "libnccl.so*" 2>/dev/null
/usr/local/lib/python3.12/dist-packages/nvidia/nccl/lib/libnccl.so.2
/usr/lib/x86_64-linux-gnu/libnccl.so
/usr/lib/x86_64-linux-gnu/libnccl.so.2
/usr/lib/x86_64-linux-gnu/libnccl.so.2.26.5
root@5468a704:/workspace/vllm-0.9.2# 

At first, we try to run this files examples/online_serving/disaggregated_prefill.sh. ,

#!/bin/bash
# This file demonstrates the example usage of disaggregated prefilling
# We will launch 2 vllm instances (1 for prefill and 1 for decode),
# and then transfer the KV cache between them.

set -xe

echo "🚧🚧 Warning: The usage of disaggregated prefill is experimental and subject to change 🚧🚧"
sleep 1

# meta-llama/Meta-Llama-3.1-8B-Instruct or deepseek-ai/DeepSeek-V2-Lite
MODEL_NAME=${MODEL:-/mnt/LLM_models/llama3-8b-instruct}
# MODEL_NAME=${MODEL:-/mnt/LLM_models/Qwen3-32B}

# Trap the SIGINT signal (triggered by Ctrl+C)
trap 'cleanup' INT

# Cleanup function
cleanup() {
    echo "Caught Ctrl+C, cleaning up..."
    # Cleanup commands
    pgrep python | xargs kill -9
    pkill -f python
    echo "Cleanup complete. Exiting."
    exit 0
}

export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')

# install quart first -- required for disagg prefill proxy serve
if python3 -c "import quart" &> /dev/null; then
    echo "Quart is already installed."
else
    echo "Quart is not installed. Installing..."
    python3 -m pip install quart
fi 

# a function that waits vLLM server to start
wait_for_server() {
  local port=$1
  timeout 1200 bash -c "
    until curl -s localhost:${port}/v1/completions > /dev/null; do
      sleep 1
    done" && return 0 || return 1
}


# You can also adjust --kv-ip and --kv-port for distributed inference.

# prefilling instance, which is the KV producer
# CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME \
#     --port 8100 \
#     --max-model-len 100 \
#     --gpu-memory-utilization 0.6 \
#     --trust-remote-code \
#     --tensor-parallel-size 4 \
#     --kv-transfer-config \
#     '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}' &

#? change CUDA_VISIBLE_DEVICES to 0,1,2,3 if you want to use more GPUs

CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME \
    --port 8100 \
    --max-model-len 100 \
    --gpu-memory-utilization 0.6 \
    --trust-remote-code \
    --kv-transfer-config \
    '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}' &

# decoding instance, which is the KV consumer
# CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME \
#     --port 8200 \
#     --max-model-len 100 \
#     --gpu-memory-utilization 0.6 \
#     --trust-remote-code \
#     --tensor-parallel-size 4 \
#     --kv-transfer-config \
#     '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}' &

#? change CUDA_VISIBLE_DEVICES to 4,5,6,7 if you want to use more GPUs

CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME \
    --port 8200 \
    --max-model-len 100 \
    --gpu-memory-utilization 0.6 \
    --trust-remote-code \
    --kv-transfer-config \
    '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}' &

# wait until prefill and decode instances are ready
wait_for_server 8100
wait_for_server 8200

# launch a proxy server that opens the service at port 8000
# the workflow of this proxy:
# - send the request to prefill vLLM instance (port 8100), change max_tokens 
#   to 1
# - after the prefill vLLM finishes prefill, send the request to decode vLLM 
#   instance
# NOTE: the usage of this API is subject to change --- in the future we will 
# introduce "vllm connect" to connect between prefill and decode instances
python3 ../../benchmarks/disagg_benchmarks/disagg_prefill_proxy_server.py &
sleep 1

# serve two example requests
output1=$(curl -X POST -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "'"$MODEL_NAME"'",
"prompt": "San Francisco is a",
"max_tokens": 10,
"temperature": 0
}')

output2=$(curl -X POST -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "'"$MODEL_NAME"'",
"prompt": "Santa Clara is a",
"max_tokens": 10,
"temperature": 0
}')


# Cleanup commands
pgrep python | xargs kill -9
pkill -f python

echo ""

sleep 1

# Print the outputs of the curl requests
echo ""
echo "Output of first request: $output1"
echo "Output of second request: $output2"

echo "🎉🎉 Successfully finished 2 test requests! 🎉🎉"
echo `""`

But , We get this error:

root@5468a704:/workspace/vllm-0.9.2/examples/online_serving# bash disaggregated_prefill.sh 
+ echo '🚧🚧 Warning: The usage of disaggregated prefill is experimental and subject to change 🚧🚧'
🚧🚧 Warning: The usage of disaggregated prefill is experimental and subject to change 🚧🚧
+ sleep 1
+ MODEL_NAME=/mnt/LLM_models/llama3-8b-instruct
+ trap cleanup INT
++ hostname -I
++ awk '{print $1}'
+ export VLLM_HOST_IP=10.66.6.76
+ VLLM_HOST_IP=10.66.6.76
+ python3 -c 'import quart'
+ echo 'Quart is already installed.'
Quart is already installed.
+ CUDA_VISIBLE_DEVICES=0
+ vllm serve /mnt/LLM_models/llama3-8b-instruct --port 8100 --max-model-len 100 --gpu-memory-utilization 0.6 --trust-remote-code --kv-transfer-config '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}'
+ wait_for_server 8100
+ local port=8100
+ timeout 1200 bash -c '
    until curl -s localhost:8100/v1/completions > /dev/null; do
      sleep 1
    done'
+ CUDA_VISIBLE_DEVICES=1
+ vllm serve /mnt/LLM_models/llama3-8b-instruct --port 8200 --max-model-len 100 --gpu-memory-utilization 0.6 --trust-remote-code --kv-transfer-config '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}'
INFO 07-30 09:34:33 [__init__.py:235] Automatically detected platform cuda.
INFO 07-30 09:34:33 [__init__.py:235] Automatically detected platform cuda.
INFO 07-30 09:34:35 [api_server.py:1755] vLLM API server version 0.10.0
INFO 07-30 09:34:35 [cli_args.py:261] non-default args: {'model_tag': '/mnt/LLM_models/llama3-8b-instruct', 'port': 8100, 'model': '/mnt/LLM_models/llama3-8b-instruct', 'trust_remote_code': True, 'max_model_len': 100, 'gpu_memory_utilization': 0.6, 'kv_transfer_config': KVTransferConfig(kv_connector='PyNcclConnector', engine_id='1103cc76-1467-4ae0-82d4-f1ee50f2d573', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_producer', kv_rank=0, kv_parallel_size=2, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={}, kv_connector_module_path=None)}
INFO 07-30 09:34:35 [api_server.py:1755] vLLM API server version 0.10.0
INFO 07-30 09:34:35 [cli_args.py:261] non-default args: {'model_tag': '/mnt/LLM_models/llama3-8b-instruct', 'port': 8200, 'model': '/mnt/LLM_models/llama3-8b-instruct', 'trust_remote_code': True, 'max_model_len': 100, 'gpu_memory_utilization': 0.6, 'kv_transfer_config': KVTransferConfig(kv_connector='PyNcclConnector', engine_id='456b0445-1d41-48cc-9fea-dc66257ef875', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_consumer', kv_rank=1, kv_parallel_size=2, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={}, kv_connector_module_path=None)}
INFO 07-30 09:34:42 [config.py:1604] Using max model len 100
INFO 07-30 09:34:42 [config.py:1604] Using max model len 100
INFO 07-30 09:34:42 [config.py:2434] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 07-30 09:34:42 [config.py:2434] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 07-30 09:34:47 [__init__.py:235] Automatically detected platform cuda.
INFO 07-30 09:34:47 [__init__.py:235] Automatically detected platform cuda.
INFO 07-30 09:34:49 [core.py:572] Waiting for init message from front-end.
INFO 07-30 09:34:49 [core.py:71] Initializing a V1 LLM engine (v0.10.0) with config: model='/mnt/LLM_models/llama3-8b-instruct', speculative_config=None, tokenizer='/mnt/LLM_models/llama3-8b-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=100, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/mnt/LLM_models/llama3-8b-instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
INFO 07-30 09:34:49 [core.py:572] Waiting for init message from front-end.
INFO 07-30 09:34:49 [core.py:71] Initializing a V1 LLM engine (v0.10.0) with config: model='/mnt/LLM_models/llama3-8b-instruct', speculative_config=None, tokenizer='/mnt/LLM_models/llama3-8b-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=100, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/mnt/LLM_models/llama3-8b-instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
[W730 09:35:01.518286809 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W730 09:35:01.618087217 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W730 09:35:11.528850450 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
INFO 07-30 09:35:11 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
ERROR 07-30 09:35:11 [core.py:632] EngineCore failed to start.
ERROR 07-30 09:35:11 [core.py:632] Traceback (most recent call last):
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
ERROR 07-30 09:35:11 [core.py:632]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 07-30 09:35:11 [core.py:632]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in __init__
ERROR 07-30 09:35:11 [core.py:632]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 77, in __init__
ERROR 07-30 09:35:11 [core.py:632]     self.model_executor = executor_class(vllm_config)
ERROR 07-30 09:35:11 [core.py:632]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
ERROR 07-30 09:35:11 [core.py:632]     self._init_executor()
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
ERROR 07-30 09:35:11 [core.py:632]     self.collective_rpc("init_device")
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
ERROR 07-30 09:35:11 [core.py:632]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-30 09:35:11 [core.py:632]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
ERROR 07-30 09:35:11 [core.py:632]     return func(*args, **kwargs)
ERROR 07-30 09:35:11 [core.py:632]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
ERROR 07-30 09:35:11 [core.py:632]     self.worker.init_device()  # type: ignore
ERROR 07-30 09:35:11 [core.py:632]     ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
ERROR 07-30 09:35:11 [core.py:632]     init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 584, in init_worker_distributed_environment
ERROR 07-30 09:35:11 [core.py:632]     ensure_kv_transfer_initialized(vllm_config)
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 64, in ensure_kv_transfer_initialized
ERROR 07-30 09:35:11 [core.py:632]     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
ERROR 07-30 09:35:11 [core.py:632]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
ERROR 07-30 09:35:11 [core.py:632]     assert issubclass(connector_cls, KVConnectorBase_V1)
ERROR 07-30 09:35:11 [core.py:632]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632] AssertionError
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 636, in run_engine_core
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 77, in __init__
    self.model_executor = executor_class(vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
    self.collective_rpc("init_device")
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
    self.worker.init_device()  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
    init_worker_distributed_environment(self.vllm_config, self.rank,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 584, in init_worker_distributed_environment
    ensure_kv_transfer_initialized(vllm_config)
  File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 64, in ensure_kv_transfer_initialized
    _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
    assert issubclass(connector_cls, KVConnectorBase_V1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
[W730 09:35:11.628729904 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
INFO 07-30 09:35:11 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
ERROR 07-30 09:35:11 [core.py:632] EngineCore failed to start.
ERROR 07-30 09:35:11 [core.py:632] Traceback (most recent call last):
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
ERROR 07-30 09:35:11 [core.py:632]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 07-30 09:35:11 [core.py:632]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in __init__
ERROR 07-30 09:35:11 [core.py:632]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 77, in __init__
ERROR 07-30 09:35:11 [core.py:632]     self.model_executor = executor_class(vllm_config)
ERROR 07-30 09:35:11 [core.py:632]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
ERROR 07-30 09:35:11 [core.py:632]     self._init_executor()
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
ERROR 07-30 09:35:11 [core.py:632]     self.collective_rpc("init_device")
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
ERROR 07-30 09:35:11 [core.py:632]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-30 09:35:11 [core.py:632]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
ERROR 07-30 09:35:11 [core.py:632]     return func(*args, **kwargs)
ERROR 07-30 09:35:11 [core.py:632]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
ERROR 07-30 09:35:11 [core.py:632]     self.worker.init_device()  # type: ignore
ERROR 07-30 09:35:11 [core.py:632]     ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
ERROR 07-30 09:35:11 [core.py:632]     init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 584, in init_worker_distributed_environment
ERROR 07-30 09:35:11 [core.py:632]     ensure_kv_transfer_initialized(vllm_config)
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 64, in ensure_kv_transfer_initialized
ERROR 07-30 09:35:11 [core.py:632]     _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
ERROR 07-30 09:35:11 [core.py:632]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
ERROR 07-30 09:35:11 [core.py:632]     assert issubclass(connector_cls, KVConnectorBase_V1)
ERROR 07-30 09:35:11 [core.py:632]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 09:35:11 [core.py:632] AssertionError
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 636, in run_engine_core
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 77, in __init__
    self.model_executor = executor_class(vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
    self.collective_rpc("init_device")
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
    self.worker.init_device()  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
    init_worker_distributed_environment(self.vllm_config, self.rank,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 584, in init_worker_distributed_environment
    ensure_kv_transfer_initialized(vllm_config)
  File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 64, in ensure_kv_transfer_initialized
    _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v1(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 73, in create_connector_v1
    assert issubclass(connector_cls, KVConnectorBase_V1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
[rank0]:[W730 09:35:12.059828109 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[rank0]:[W730 09:35:12.193948453 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 52, in cmd
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1791, in run_server
    await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1811, in run_server_worker
    async with build_async_engine_client(args, client_config) as engine_client:
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 163, in from_vllm_config
    return cls(
           ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 117, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 98, in make_async_mp_client
    return AsyncMPClient(*client_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 677, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 408, in __init__
    with launch_core_engines(vllm_config, executor_class,
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 697, in launch_core_engines
    wait_for_engine_startup(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 52, in cmd
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1791, in run_server
    await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1811, in run_server_worker
    async with build_async_engine_client(args, client_config) as engine_client:
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 163, in from_vllm_config
    return cls(
           ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 117, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 98, in make_async_mp_client
    return AsyncMPClient(*client_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 677, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 408, in __init__
    with launch_core_engines(vllm_config, executor_class,
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 697, in launch_core_engines
    wait_for_engine_startup(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
^C^C^C^C++ cleanup
++ echo 'Caught Ctrl+C, cleaning up...'
Caught Ctrl+C, cleaning up...
++ pgrep python
++ xargs kill -9

Usage:
 kill [options] <pid> [...]

Options:
 <pid> [...]            send signal to every <pid> listed
 -<signal>, -s, --signal <signal>
                        specify the <signal> to be sent
 -q, --queue <value>    integer value to be sent with the signal
 -l, --list=[<signal>]  list all signal names, or convert one to a name
 -L, --table            list all signal names in a nice table

 -h, --help     display this help and exit
 -V, --version  output version information and exit

For more details see kill(1).

[Alan-D-Chen]

We get 8 GPUs in the server. We have updated the vllm.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L20                     On  |   00000000:23:00.0 Off |                  Off |
| N/A   32C    P8             35W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L20                     On  |   00000000:24:00.0 Off |                  Off |
| N/A   33C    P8             35W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L20                     On  |   00000000:25:00.0 Off |                  Off |
| N/A   33C    P8             35W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L20                     On  |   00000000:27:00.0 Off |                  Off |
| N/A   33C    P0             67W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA L20                     On  |   00000000:84:00.0 Off |                  Off |
| N/A   32C    P8             36W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA L20                     On  |   00000000:85:00.0 Off |                  Off |
| N/A   34C    P0             52W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA L20                     On  |   00000000:86:00.0 Off |                  Off |
| N/A   33C    P8             36W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA L20                     On  |   00000000:87:00.0 Off |                  Off |
| N/A   33C    P8             64W /  350W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

And we want to run Qwen3-32B PD disaggregation with 4 GPUs for prefill and 4 GPUs for decode in the
server node 76. Then in the future, we wanna run Qwen3-32B PD disaggregation with 4 GPUs for prefill and 4 GPUs for decode in server node 76 and with 4 GPUs (in group ONE) for prefill ,with 4 GPUs (in Group TWO) for prefill in other servers.

当前问题的核心是 vllm 0.10.0 中 PyNcclConnector 的实现缺陷，导致多 GPU 场景下断言失败。最有效的解决方式是升级到更高版本的 vllm（如 0.11.0+），或通过降低张量并行规模、修改源码等方式规避。由于分离式预填充是实验性功能，建议关注官方后续版本的修复进展。

However, the core issue remains unresolved. The root cause of the current problem is a flaw in the implementation of PyNcclConnector in vllm 0.10.0, which leads to an assertion failure in multi-GPU scenarios. The most effective solutions are to upgrade to a newer version of vllm (such as 0.11.0 or later), or to work around it by reducing the tensor parallel size, modifying the source code, etc.

We sincerely hope to receive your help. Thank you again.

[Alan-D-Chen]

# You can also adjust --kv-ip and --kv-port for distributed inference.

# prefilling instance, which is the KV producer
# CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve $MODEL_NAME \
#     --port 8100 \
#     --max-model-len 100 \
#     --gpu-memory-utilization 0.6 \
#     --trust-remote-code \
#     --tensor-parallel-size 4 \
#     --kv-transfer-config \
#     '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}' &

#? change CUDA_VISIBLE_DEVICES to 0,1,2,3 if you want to use more GPUs


# decoding instance, which is the KV consumer
# CUDA_VISIBLE_DEVICES=4,5,6,7 vllm serve $MODEL_NAME \
#     --port 8200 \
#     --max-model-len 100 \
#     --gpu-memory-utilization 0.6 \
#     --trust-remote-code \
#     --tensor-parallel-size 4 \
#     --kv-transfer-config \
#     '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}' &

#? change CUDA_VISIBLE_DEVICES to 4,5,6,7 if you want to use more GPUs

[jikunshang]

PyNcclConnector is not supported in v1, please switch to nixl connector or lmcache connector instead. if you really want to use PyNcclConnector, you can fallback to V0 with VLLM_USE_V1=0, while this would be removed in near future.

[frankie-ys]

I meet another question， I have 2 pieces GPU at one docker container， I find that when I start the prefill instance and start the decode instance at the same time, it's always hanged at selecting device cuda ,which I config that

#prefilling instance
CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME
--port 8100
--max-model-len 100
--gpu-memory-utilization 0.6
--trust-remote-code
--tensor-parallel-size 1
--kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}'

decoding instance

CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME
--port 8200
--max-model-len 100
--gpu-memory-utilization 0.6
--trust-remote-code
--tensor-parallel-size 1
--kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}'

When I use config which kv_parallel_size is set to 1, it can start the prefill instance but get error when start decode instance. the error is assert error, that self.rank < self.world_size ,and the params like this
#prefill instance
CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME
--port 8100
--max-model-len 100
--gpu-memory-utilization 0.6
--trust-remote-code
--tensor-parallel-size 1
--kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":1}'

decoding instance

CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME
--port 8200
--max-model-len 100
--gpu-memory-utilization 0.6
--trust-remote-code
--tensor-parallel-size 1
--kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":1}' , I'd thanks very much if any developer can help me solve this problem!

[frankie-ys]

I meet another question， I have 2 pieces GPU at one docker container， I find that when I start the prefill instance and start the decode instance at the same time, it's always hanged at selecting device cuda ,which I config that

#prefilling instance CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME --port 8100 --max-model-len 100 --gpu-memory-utilization 0.6 --trust-remote-code --tensor-parallel-size 1 --kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}'

decoding instance

CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME --port 8200 --max-model-len 100 --gpu-memory-utilization 0.6 --trust-remote-code --tensor-parallel-size 1 --kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}'

When I use config which kv_parallel_size is set to 1, it can start the prefill instance but get error when start decode instance. the error is assert error, that self.rank < self.world_size ,and the params like this #prefill instance CUDA_VISIBLE_DEVICES=0 vllm serve $MODEL_NAME --port 8100 --max-model-len 100 --gpu-memory-utilization 0.6 --trust-remote-code --tensor-parallel-size 1 --kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":1}'

decoding instance

CUDA_VISIBLE_DEVICES=1 vllm serve $MODEL_NAME --port 8200 --max-model-len 100 --gpu-memory-utilization 0.6 --trust-remote-code --tensor-parallel-size 1 --kv-transfer-config \ '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":1}' , I'd thanks very much if any developer can help me solve this problem!

I find that it can be worked in vllm 0.8.5

[Alan-D-Chen]

@frankie-ys dear friend, we wanna run Qwen3-32B PD disaggregation with 4 GPUs for prefill and 4 GPUs for decode in server node 76 and with 4 GPUs (in group ONE) for prefill ,with 4 GPUs (in Group TWO) for prefill in other servers. Can you do this ?