vLLM segfault when serving Qwen3.8-27B with image: intel/llm-scaler-vllm:0.21.0-b3.1

[noobHappylife]

When using the latest docker image intel/llm-scaler-vllm:0.21.0-b3.1, when on a 4x B60 machine serving Qwen3.8-27B when compile and graph mode is on, it will have segfault after torch compile. I only can serve it with --enforce-eager.

image used:
intel/llm-scaler-vllm:0.21.0-b3.1

serve command:
VLLM_XPU_ENABLE_XPU_GRAPH=1 vllm serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --mm-encoder-tp-mode data --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --port 8020 --quantization sym_int4 --dtype float16 --block-size 64 --trust-remote-code

error logs:

(Worker_TP0 pid=6509) INFO 08-20 01:58:15 [backends.py:1148] Dynamo bytecode transform time: 0.55 s
(Worker_TP2 pid=6511) WARNING 08-20 01:58:15 [backends.py:1048] Failed to read file <frozen os>
(Worker_TP3 pid=6512) WARNING 08-20 01:58:15 [backends.py:1048] Failed to read file <frozen os>
(Worker_TP0 pid=6509) INFO 08-20 01:58:29 [backends.py:393] Compiling a graph for compile range (1, 2048) takes 13.89 s
(Worker_TP0 pid=6509) INFO 08-20 01:58:29 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/c90e1a6a7be3a2b56e0ef778e8227f3418fea9c6cb06508f986ac0e0210f8670/rank_0_0/model
(Worker_TP0 pid=6509) INFO 08-20 01:58:29 [monitor.py:53] torch.compile took 14.75 s in total

!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in sycl::_V1::detail::queue_impl::submit_impl(sycl::_V1::detail::type_erased_cgfo_ty const&, sycl::_V1::detail::queue_impl*, bool, sycl::_V1::detail::code_location const&, bool, sycl::_V1::detail::v1::SubmissionInfo const&)
  File "<unknown>", line 0, in sycl::_V1::detail::queue_impl::submit_with_event(sycl::_V1::detail::type_erased_cgfo_ty const&, sycl::_V1::detail::v1::SubmissionInfo const&, sycl::_V1::detail::code_location const&, bool)
  File "<unknown>", line 0, in sycl::_V1::queue::submit_with_event_impl(sycl::_V1::detail::type_erased_cgfo_ty const&, sycl::_V1::detail::v1::SubmissionInfo const&, sycl::_V1::detail::code_location const&, bool) const
  File "<unknown>", line 0, in sycl::_V1::event sycl::_V1::queue::submit_with_event<false, sycl::_V1::ext::oneapi::experimental::properties<sycl::_V1::ext::oneapi::experimental::detail::properties_type_list<> > >(sycl::_V1::ext::oneapi::experimental::properties<sycl::_V1::ext::oneapi::experimental::detail::properties_type_list<> >, sycl::_V1::detail::type_erased_cgfo_ty const&, sycl::_V1::detail::code_location const&) const
  File "<unknown>", line 0, in invoke_barrier(std::shared_ptr<ccl_comm>, sycl::_V1::queue, std::vector<sycl::_V1::event, std::allocator<sycl::_V1::event> > const&, bool)
  File "<unknown>", line 0, in sycl::_V1::event allgatherv_large_su_ring<sycl::_V1::detail::half_impl::half>(sycl::_V1::queue&, void const*, unsigned long, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, sycl_ptrs_type&, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&)
  File "<unknown>", line 0, in ccl::v1::event invoke_collective_type<4, 1, allgatherv_large(sycl::_V1::queue&, void const*, unsigned long, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool)::{lambda<typename $T0, int $N1, int $N2>()#1}>(allgatherv_large(sycl::_V1::queue&, void const*, unsigned lo
ng, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool)::{lambda<typename $T0, int $N1, int $N2>()#1}, ccl::v1::datatype)
  File "<unknown>", line 0, in ccl::v1::event invoke_collective_size<1, allgatherv_large(sycl::_V1::queue&, void const*, unsigned long, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool)::{lambda<typename $T0, int $N1, int $N2>()#1}>(allgatherv_large(sycl::_V1::queue&, void const*, unsigned long,
 void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool)::{lambda<typename $T0, int $N1, int $N2>()#1}, int, ccl::v1::datatype)
  File "<unknown>", line 0, in allgatherv_large(sycl::_V1::queue&, void const*, unsigned long, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool)
  File "<unknown>", line 0, in ccl::v1::allgather_sycl_single_node(sycl::_V1::queue&, void const*, unsigned long, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool&, bool)
  File "<unknown>", line 0, in ccl::v1::allgather_sycl(sycl::_V1::queue&, void const*, unsigned long, void*, std::vector<unsigned long, std::allocator<unsigned long> > const&, ccl::v1::datatype, ccl_comm*, ccl_stream*, ccl::v1::allgatherv_attr const&, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&, bool&)
  File "<unknown>", line 0, in std::_Function_handler<ccl::v1::event (), ccl_allgather(void const*, void*, unsigned long, ccl::v1::datatype, ccl_coll_attr const&, ccl_comm*, ccl_stream const*, std::vector<ccl::v1::event, std::allocator<ccl::v1::event> > const&)::{lambda()#2}>::_M_invoke(std::_Any_data const&)
  File "<unknown>", line 0, in group_impl::end()
  File "<unknown>", line 0, in c10d::ProcessGroupXCCL::allgather_into_tensor_coalesced(std::vector<at::Tensor, std::allocator<at::Tensor> >&, std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllgatherOptions const&)
  File "<unknown>", line 0, in c10d::ops::(anonymous namespace)::allgather_into_tensor_coalesced_XPU(c10::ArrayRef<at::Tensor>, c10::ArrayRef<at::Tensor>, c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, bool)
  File "<unknown>", line 0, in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<c10::intrusive_ptr<c10d::Work, c10::detail::intrusive_target_default_null_type<c10d::Work> > (*)(c10::ArrayRef<at::Tensor>, c10::ArrayRef<at::Tensor>, c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, bool), c10::intrusive_ptr<c10d::Work, c10::detail::intrusive_target_default_null_type<c10d::Work> >, c10::guts::typ
elist::typelist<c10::ArrayRef<at::Tensor>, c10::ArrayRef<at::Tensor>, c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, bool> >, c10::intrusive_ptr<c10d::Work, c10::detail::intrusive_target_default_null_type<c10d::Work> > (c10::ArrayRef<at::Tensor>, c10::ArrayRef<at::Tensor>, c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, bool)>::call(c10::OperatorKernel*, c10::Disp
atchKeySet, c10::ArrayRef<at::Tensor>, c10::ArrayRef<at::Tensor>, c10::intrusive_ptr<c10d::ProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::ProcessGroup> > const&, bool)
  File "<unknown>", line 0, in c10d::ProcessGroup::allgather_into_tensor_coalesced(std::vector<at::Tensor, std::allocator<at::Tensor> >&, std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllgatherOptions const&)
  File "<unknown>", line 0, in c10d::all_gather_into_tensor_coalesced(std::vector<at::Tensor, std::allocator<at::Tensor> >, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
  File "<unknown>", line 0, in c10d::all_gather_into_tensor(at::Tensor const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
  File "<unknown>", line 0, in c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<at::Tensor (*)(at::Tensor const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, false>::call(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValu
e, std::allocator<c10::IValue> >*)
  File "<unknown>", line 0, in c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const [clone .isra.0]
  File "<unknown>", line 0, in torch::jit::invokeOperatorFromPython(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, pybind11::args const&, pybind11::kwargs const&, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in torch::jit::_get_operation_for_overload_or_packet(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::
args const&, pybind11::kwargs const&)#1}, pybind11::object, pybind11::args const&, pybind11::kwargs const&>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const&) const::{lambda(pybind11::args const&, pybind11::kwargs const&)#1}&&, pybind11::object (*)(pybind11::args const&, pybind11::kwargs const&))::{lambda(pybind11::detail::function_call&)#1}::_FUN(pybind11::detail::function_call&)
  File "<unknown>", line 0, in pybind11::cpp_function::dispatcher(_object*, _object*, _object*)
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in PyObject_CallMethod
  File "<unknown>", line 0, in torch::handle_torch_function_no_python_arg_parser(c10::ArrayRef<_object*>, _object*, _object*, char const*, _object*, char const*, c10::OperatorHandle const*, std::vector<c10::IValue, std::allocator<c10::IValue> >*, torch::TorchFunctionName)
  File "<unknown>", line 0, in torch::detail::(anonymous namespace)::ConcretePyInterpreterVTable::dispatch(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const
  File "<unknown>", line 0, in (anonymous namespace)::pythonFallback(c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*)
  File "<unknown>", line 0, in void c10::BoxedKernel::make_boxed_function<&(anonymous namespace)::pythonTLSSnapshotFallback>(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*)
  File "<unknown>", line 0, in c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const [clone .isra.0]
  File "<unknown>", line 0, in torch::jit::invokeOperatorFromPython(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, pybind11::args const&, pybind11::kwargs const&, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in torch::jit::_get_operation_for_overload_or_packet(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::
args const&, pybind11::kwargs const&)#1}, pybind11::object, pybind11::args const&, pybind11::kwargs const&>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator
<char> > const&) const::{lambda(pybind11::args const&, pybind11::kwargs const&)#1}&&, pybind11::object (*)(pybind11::args const&, pybind11::kwargs const&))::{lambda(pybind11::detail::function_call&)#1}::_FUN(pybind11::detail::function_call&)
  File "<unknown>", line 0, in pybind11::cpp_function::dispatcher(_object*, _object*, _object*)
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in PyEval_EvalCode
  File "<unknown>", line 0, in PyRun_StringFlags
  File "<unknown>", line 0, in PyRun_SimpleStringFlags
  File "<unknown>", line 0, in Py_RunMain
  File "<unknown>", line 0, in Py_BytesMain
  File "<unknown>", line 0, in _start
  File "<unknown>", line 0, in 0xffffffffffffffff

``

[365614269]

Reproduced exactly the same problem, suspect it's enable-xpu-graph's problem. Investigating on this, will catch up later

[365614269]

Also served it successfully with --enforce-eager, genuinely looks like xpu-graph.

By the way, From the 0.21.0-b1 release notes:

Upgrade to vLLM 0.21.0
Experimentally support XPU graph

XPU graph support was introduced experimentally in the 0.21.0 line, and --enforce-eager is exactly the flag that turns it off. Your OOM appeared in the first release series where this path is enabled by default, and disabling it fixed the problem.

So: --enforce-eager is the correct workaround for this image, I'd say keep it for a while.

[dominick253]

I also get the same issue.