[Bug]: Failed to deploy Qwen3-235B-A22B-Thinking-2507 with 0.9.2rc1 #2117
Description
Your current environment
The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.5.1
Is debug build: False
OS: Ubuntu 22.04.5 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 4.0.3
Libc version: glibc-2.35
Python version: 3.10.17 (main, May 8 2025, 07:18:04) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.0-182.0.0.95.r1941_123.hce2.aarch64-aarch64-with-glibc2.35
CPU:
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 320
On-line CPU(s) list: 0-319
Vendor ID: HiSilicon
BIOS Vendor ID: HiSilicon
BIOS Model name: Kunpeng 920 7285Z
Model: 0
Thread(s) per core: 1
Core(s) per socket: 80
Socket(s): 4
Stepping: 0x0
Frequency boost: disabled
CPU max MHz: 3000.0000
CPU min MHz: 400.0000
BogoMIPS: 200.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint svei8mm svef32mm svef64mm svebf16 i8mm bf16 dgh rng ecv
L1d cache: 20 MiB (320 instances)
L1i cache: 20 MiB (320 instances)
L2 cache: 400 MiB (320 instances)
L3 cache: 560 MiB (8 instances)
NUMA node(s): 8
NUMA node0 CPU(s): 0-39
NUMA node1 CPU(s): 40-79
NUMA node2 CPU(s): 80-119
NUMA node3 CPU(s): 120-159
NUMA node4 CPU(s): 160-199
NUMA node5 CPU(s): 200-239
NUMA node6 CPU(s): 240-279
NUMA node7 CPU(s): 280-319
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==27.0.0
[pip3] torch==2.5.1
[pip3] torch-npu==2.5.1.post1.dev20250619
[pip3] torchvision==0.20.1
[pip3] transformers==4.52.4
[conda] Could not collect
vLLM Version: 0.9.2
vLLM Ascend Version: 0.9.2rc1
ENV Variables:
ATB_OPSRUNNER_KERNEL_CACHE_TILING_SIZE=10240
ATB_OPSRUNNER_KERNEL_CACHE_LOCAL_COUNT=1
ATB_STREAM_SYNC_EVERY_RUNNER_ENABLE=0
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
ATB_OPSRUNNER_SETUP_CACHE_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_GLOBAL=0
ATB_DEVICE_TILING_BUFFER_BLOCK_NUM=32
ATB_STREAM_SYNC_EVERY_KERNEL_ENABLE=0
ATB_OPSRUNNER_KERNEL_CACHE_GLOABL_COUNT=5
ATB_HOME_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0
ASCEND_TOOLKIT_HOME=/usr/local/Ascend/ascend-toolkit/latest
ATB_COMPARE_TILING_EVERY_KERNEL=0
ASCEND_OPP_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp
LD_LIBRARY_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/Ascend/driver/lib64/common/:/usr/local/Ascend/driver/lib64/driver/:
ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_OPSRUNNER_KERNEL_CACHE_TYPE=3
ATB_RUNNER_POOL_SIZE=64
ATB_STREAM_SYNC_EVERY_OPERATION_ENABLE=0
ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_MATMUL_SHUFFLE_K_ENABLE=1
ATB_LAUNCH_KERNEL_WITH_TILING=1
ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=1
ATB_HOST_TILING_BUFFER_BLOCK_NUM=128
ATB_SHARE_MEMORY_NAME_SUFFIX=
TORCH_DEVICE_BACKEND_AUTOLOAD=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
NPU:
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3.5 Version: 24.1.rc3.5 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip Phy-ID | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 Ascend910 | OK | 185.5 36 0 / 0 |
| 0 0 | 0000:9D:00.0 | 0 0 / 0 3407 / 65536 |
+------------------------------------------------------------------------------------------------+
| 0 Ascend910 | OK | - 37 0 / 0 |
| 1 1 | 0000:9F:00.0 | 0 0 / 0 3186 / 65536 |
+===========================+===============+====================================================+
| 1 Ascend910 | OK | 182.4 36 0 / 0 |
| 0 2 | 0000:99:00.0 | 0 0 / 0 3396 / 65536 |
+------------------------------------------------------------------------------------------------+
| 1 Ascend910 | OK | - 36 0 / 0 |
| 1 3 | 0000:9B:00.0 | 0 0 / 0 3196 / 65536 |
+===========================+===============+====================================================+
| 2 Ascend910 | OK | 182.3 35 0 / 0 |
| 0 4 | 0000:95:00.0 | 0 0 / 0 3391 / 65536 |
+------------------------------------------------------------------------------------------------+
| 2 Ascend910 | OK | - 37 0 / 0 |
| 1 5 | 0000:97:00.0 | 0 0 / 0 3199 / 65536 |
+===========================+===============+====================================================+
| 3 Ascend910 | OK | 180.8 37 0 / 0 |
| 0 6 | 0000:91:00.0 | 0 0 / 0 3404 / 65536 |
+------------------------------------------------------------------------------------------------+
| 3 Ascend910 | OK | - 36 0 / 0 |
| 1 7 | 0000:93:00.0 | 0 0 / 0 3185 / 65536 |
+===========================+===============+====================================================+
| 4 Ascend910 | OK | 182.9 37 0 / 0 |
| 0 8 | 0000:8D:00.0 | 0 0 / 0 3396 / 65536 |
+------------------------------------------------------------------------------------------------+
| 4 Ascend910 | OK | - 36 0 / 0 |
| 1 9 | 0000:8F:00.0 | 0 0 / 0 3196 / 65536 |
+===========================+===============+====================================================+
| 5 Ascend910 | OK | 183.7 35 0 / 0 |
| 0 10 | 0000:89:00.0 | 0 0 / 0 3392 / 65536 |
+------------------------------------------------------------------------------------------------+
| 5 Ascend910 | OK | - 36 0 / 0 |
| 1 11 | 0000:8B:00.0 | 0 0 / 0 3199 / 65536 |
+===========================+===============+====================================================+
| 6 Ascend910 | OK | 185.1 36 0 / 0 |
| 0 12 | 0000:85:00.0 | 0 0 / 0 3392 / 65536 |
+------------------------------------------------------------------------------------------------+
| 6 Ascend910 | OK | - 37 0 / 0 |
| 1 13 | 0000:87:00.0 | 0 0 / 0 3198 / 65536 |
+===========================+===============+====================================================+
| 7 Ascend910 | OK | 193.0 37 0 / 0 |
| 0 14 | 0000:81:00.0 | 0 0 / 0 3406 / 65536 |
+------------------------------------------------------------------------------------------------+
| 7 Ascend910 | OK | - 36 0 / 0 |
| 1 15 | 0000:83:00.0 | 0 0 / 0 3187 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| No running processes found in NPU 0 |
+===========================+===============+====================================================+
| No running processes found in NPU 1 |
+===========================+===============+====================================================+
| No running processes found in NPU 2 |
+===========================+===============+====================================================+
| No running processes found in NPU 3 |
+===========================+===============+====================================================+
| No running processes found in NPU 4 |
+===========================+===============+====================================================+
| No running processes found in NPU 5 |
+===========================+===============+====================================================+
| No running processes found in NPU 6 |
+===========================+===============+====================================================+
| No running processes found in NPU 7 |
+===========================+===============+====================================================+
CANN:
package_name=Ascend-cann-toolkit
version=8.1.RC1
innerversion=V100R001C21SPC001B238
compatible_version=[V100R001C15],[V100R001C18],[V100R001C19],[V100R001C20],[V100R001C21]
arch=aarch64
os=linux
path=/usr/local/Ascend/ascend-toolkit/8.1.RC1/aarch64-linux
🐛 Describe the bug
Failed to deploy Qwen3-235B-A22B-Thinking-2507
vllm serve /models/Qwen3-235B-A22B-Thinking-2507 \
--tensor-parallel-size 16 \
--host 0.0.0.0 \
--port 6000 \
--served-model-name chat \
--trust_remote_code \
--enable_expert_parallel \
--api-key lijun \
--disable-log-requests \
--gpu-memory-utilization 0.9 \
--max-model-len 32768Error:
(VllmWorker rank=10 pid=5568) ERROR 07-30 10:15:05 [multiproc_executor.py:487] [ERROR] 2025-07-30-10:15:05 (PID:5568, Device:10, RankID:-1) ERR01100 OPS call acl api failed
ERROR 07-30 10:15:09 [core.py:586] EngineCore failed to start.
ERROR 07-30 10:15:09 [core.py:586] Traceback (most recent call last):
ERROR 07-30 10:15:09 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core
ERROR 07-30 10:15:09 [core.py:586] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 07-30 10:15:09 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__
ERROR 07-30 10:15:09 [core.py:586] super().__init__(vllm_config, executor_class, log_stats,
ERROR 07-30 10:15:09 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 75, in __init__
ERROR 07-30 10:15:09 [core.py:586] self.model_executor = executor_class(vllm_config)
ERROR 07-30 10:15:09 [core.py:586] File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 53, in __init__
ERROR 07-30 10:15:09 [core.py:586] self._init_executor()
ERROR 07-30 10:15:09 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 93, in _init_executor
ERROR 07-30 10:15:09 [core.py:586] self.workers = WorkerProc.wait_for_ready(unready_workers)
ERROR 07-30 10:15:09 [core.py:586] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 422, in wait_for_ready
ERROR 07-30 10:15:09 [core.py:586] raise e from None
ERROR 07-30 10:15:09 [core.py:586] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 590, in run_engine_core
raise e
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 577, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 404, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 75, in __init__
self.model_executor = executor_class(vllm_config)
File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 53, in __init__
self._init_executor()
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 93, in _init_executor
self.workers = WorkerProc.wait_for_ready(unready_workers)
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 422, in wait_for_ready
raise e from None
Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
DEBUG 07-30 10:15:10 [utils.py:475] Waiting for 1 local, 0 remote core engine proc(s) to start.
Traceback (most recent call last):
File "/usr/local/python3.10.17/bin/vllm", line 8, in <module>
sys.exit(main())
File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 65, in main
args.dispatch_function(args)
File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 55, in cmd
uvloop.run(run_server(args))
File "/usr/local/python3.10.17/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/python3.10.17/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1431, in run_server
await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1451, in run_server_worker
async with build_async_engine_client(args, client_config) as engine_client:
File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 162, in from_vllm_config
return cls(
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 124, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 96, in make_async_mp_client
return AsyncMPClient(*client_args)
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 666, in __init__
super().__init__(
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 403, in __init__
with launch_core_engines(vllm_config, executor_class,
File "/usr/local/python3.10.17/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 434, in launch_core_engines
wait_for_engine_startup(
File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
[ERROR] 2025-07-30-10:15:11 (PID:5279, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception