[Bug]: GLM-4.1V-Thinking ValueError

[XiaYijun124]

Your current environment

vllm==0.9.2. the model weights from https://huggingface.co/zai-org/GLM-4.1V-9B-Thinking/tree/main

🐛 Describe the bug

bug:
ERROR 07-29 06:54:54 [core.py:588] EngineCore encountered a fatal error.
ERROR 07-29 06:54:54 [core.py:588] Traceback (most recent call last):
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 579, in run_engine_core
ERROR 07-29 06:54:54 [core.py:588] engine_core.run_busy_loop()
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 606, in run_busy_loop
ERROR 07-29 06:54:54 [core.py:588] self._process_engine_step()
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 631, in _process_engine_step
ERROR 07-29 06:54:54 [core.py:588] outputs, model_executed = self.step_fn()
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 235, in step
ERROR 07-29 06:54:54 [core.py:588] model_output = self.execute_model(scheduler_output)
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 221, in execute_model
ERROR 07-29 06:54:54 [core.py:588] raise err
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 212, in execute_model
ERROR 07-29 06:54:54 [core.py:588] return self.model_executor.execute_model(scheduler_output)
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
ERROR 07-29 06:54:54 [core.py:588] output = self.collective_rpc("execute_model",
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 07-29 06:54:54 [core.py:588] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/utils/init.py", line 2736, in run_method
ERROR 07-29 06:54:54 [core.py:588] return func(*args, **kwargs)
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-29 06:54:54 [core.py:588] return func(*args, **kwargs)
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 308, in execute_model
ERROR 07-29 06:54:54 [core.py:588] output = self.model_runner.execute_model(scheduler_output,
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-29 06:54:54 [core.py:588] return func(*args, **kwargs)
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1332, in execute_model
ERROR 07-29 06:54:54 [core.py:588] inputs_embeds = self.model.get_input_embeddings(
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/glm4_1v.py", line 1474, in get_input_embeddings
ERROR 07-29 06:54:54 [core.py:588] inputs_embeds = merge_multimodal_embeddings(
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 505, in merge_multimodal_embeddings
ERROR 07-29 06:54:54 [core.py:588] return _merge_multimodal_embeddings(
ERROR 07-29 06:54:54 [core.py:588] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 427, in _merge_multimodal_embeddings
ERROR 07-29 06:54:54 [core.py:588] raise ValueError(
ERROR 07-29 06:54:54 [core.py:588] ValueError: Attempted to assign 646 = 646 multimodal tokens to 1292 placeholders

vllm==0.9.2

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[XiaYijun124]

use vllm=0.10.0 alse have a similar error

ERROR 07-29 07:25:31 [core.py:634] EngineCore encountered a fatal error.
ERROR 07-29 07:25:31 [core.py:634] Traceback (most recent call last):
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
ERROR 07-29 07:25:31 [core.py:634] engine_core.run_busy_loop()
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
ERROR 07-29 07:25:31 [core.py:634] self._process_engine_step()
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
ERROR 07-29 07:25:31 [core.py:634] outputs, model_executed = self.step_fn()
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 267, in step
ERROR 07-29 07:25:31 [core.py:634] model_output = self.execute_model_with_error_logging(
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging
ERROR 07-29 07:25:31 [core.py:634] raise err
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging
ERROR 07-29 07:25:31 [core.py:634] return model_fn(scheduler_output)
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
ERROR 07-29 07:25:31 [core.py:634] output = self.collective_rpc("execute_model",
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
ERROR 07-29 07:25:31 [core.py:634] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/utils/init.py", line 2985, in run_method
ERROR 07-29 07:25:31 [core.py:634] return func(*args, **kwargs)
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-29 07:25:31 [core.py:634] return func(*args, **kwargs)
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
ERROR 07-29 07:25:31 [core.py:634] output = self.model_runner.execute_model(scheduler_output,
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-29 07:25:31 [core.py:634] return func(*args, **kwargs)
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1393, in execute_model
ERROR 07-29 07:25:31 [core.py:634] self._execute_mm_encoder(scheduler_output)
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1103, in _execute_mm_encoder
ERROR 07-29 07:25:31 [core.py:634] self.encoder_cache[req_id][input_id] = scatter_mm_placeholders(
ERROR 07-29 07:25:31 [core.py:634] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/utils.py", line 67, in scatter_mm_placeholders
ERROR 07-29 07:25:31 [core.py:634] placeholders[is_embed] = embeds
ERROR 07-29 07:25:31 [core.py:634] RuntimeError: shape mismatch: value tensor of shape [646, 4096] cannot be broadcast to indexing result of shape [1292, 4096]

[DarkLight1337]

cc @zRzRzRzRzRzRzR

[tyxiong23]

Similar errors observed.

[tyxiong23]

I found out that I can offline serve a GLM-4.1v-thinking, and use api calling to do video inference. However, the offline inference code has the reported bug.

[DarkLight1337]

Can you show your code?

[tyxiong23]

I just run the original code: https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py

python3 vision_language.py \
    --model-type glm4_1v \
    --modality video \
    --num-frames 32

[DarkLight1337]

Can you show the error log?

[tyxiong23]

Here is the error log

---

ERROR 07-30 11:27:19 [core.py:634] EngineCore encountered a fatal error. [17/1751]
ERROR 07-30 11:27:19 [core.py:634] Traceback (most recent call last):
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
ERROR 07-30 11:27:19 [core.py:634] engine_core.run_busy_loop()
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
ERROR 07-30 11:27:19 [core.py:634] self._process_engine_step()
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
ERROR 07-30 11:27:19 [core.py:634] outputs, model_executed = self.step_fn()
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 267, in step
ERROR 07-30 11:27:19 [core.py:634] model_output = self.execute_model_with_error_logging(
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging
ERROR 07-30 11:27:19 [core.py:634] raise err
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging
ERROR 07-30 11:27:19 [core.py:634] return model_fn(scheduler_output)
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
ERROR 07-30 11:27:19 [core.py:634] output = self.collective_rpc("execute_model",
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
ERROR 07-30 11:27:19 [core.py:634] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 2985, in run_method
ERROR 07-30 11:27:19 [core.py:634] return func(*args, **kwargs)
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-30 11:27:19 [core.py:634] return func(*args, **kwargs)
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
ERROR 07-30 11:27:19 [core.py:634] output = self.model_runner.execute_model(scheduler_output,
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-30 11:27:19 [core.py:634] return func(*args, **kwargs)
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1393, in execute_model
ERROR 07-30 11:27:19 [core.py:634] self._execute_mm_encoder(scheduler_output)
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1103, in _execute_mm_encoder
ERROR 07-30 11:27:19 [core.py:634] self.encoder_cache[req_id][input_id] = scatter_mm_placeholders(
ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/utils.py", line 67, in scatter_mm_placeholders
ERROR 07-30 11:27:19 [core.py:634] placeholders[is_embed] = embeds
ERROR 07-30 11:27:19 [core.py:634] ~~~~~~~~~~~~^^^^^^^^^^
ERROR 07-30 11:27:19 [core.py:634] RuntimeError: shape mismatch: value tensor of shape [299, 4096] cannot be broadcast to indexing result of shape [598, 4096]
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 636, in run_engine_core
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
engine_core.run_busy_loop()
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
self._process_engine_step()
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
outputs, model_executed = self.step_fn()
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 267, in step
model_output = self.execute_model_with_error_logging(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging
raise err
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging
return model_fn(scheduler_output)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
output = self.collective_rpc("execute_model",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 2985, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model
output = self.model_runner.execute_model(scheduler_output,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1393, in execute_model
self._execute_mm_encoder(scheduler_output)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1103, in _execute_mm_encoder
self.encoder_cache[req_id][input_id] = scatter_mm_placeholders(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/utils.py", line 67, in scatter_mm_placeholders
placeholders[is_embed] = embeds
~~~~~~~~~~~~^^^^^^^^^^
RuntimeError: shape mismatch: value tensor of shape [299, 4096] cannot be broadcast to indexing result of shape [598, 4096]

[XiaYijun124]

Here is the error log

ERROR 07-30 11:27:19 [core.py:634] EngineCore encountered a fatal error. [17/1751] ERROR 07-30 11:27:19 [core.py:634] Traceback (most recent call last): ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core ERROR 07-30 11:27:19 [core.py:634] engine_core.run_busy_loop() ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop ERROR 07-30 11:27:19 [core.py:634] self._process_engine_step() ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step ERROR 07-30 11:27:19 [core.py:634] outputs, model_executed = self.step_fn() ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 267, in step ERROR 07-30 11:27:19 [core.py:634] model_output = self.execute_model_with_error_logging( ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging ERROR 07-30 11:27:19 [core.py:634] raise err ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging ERROR 07-30 11:27:19 [core.py:634] return model_fn(scheduler_output) ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model ERROR 07-30 11:27:19 [core.py:634] output = self.collective_rpc("execute_model", ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc ERROR 07-30 11:27:19 [core.py:634] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 2985, in run_method ERROR 07-30 11:27:19 [core.py:634] return func(*args, **kwargs) ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 07-30 11:27:19 [core.py:634] return func(*args, **kwargs) ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model ERROR 07-30 11:27:19 [core.py:634] output = self.model_runner.execute_model(scheduler_output, ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 07-30 11:27:19 [core.py:634] return func(*args, **kwargs) ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1393, in execute_model ERROR 07-30 11:27:19 [core.py:634] self._execute_mm_encoder(scheduler_output) ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1103, in _execute_mm_encoder ERROR 07-30 11:27:19 [core.py:634] self.encoder_cache[req_id][input_id] = scatter_mm_placeholders( ERROR 07-30 11:27:19 [core.py:634] ^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/utils.py", line 67, in scatter_mm_placeholders ERROR 07-30 11:27:19 [core.py:634] placeholders[is_embed] = embeds ERROR 07-30 11:27:19 [core.py:634] ~~~~~~~~~~~~^^^^^^^^^^ ERROR 07-30 11:27:19 [core.py:634] RuntimeError: shape mismatch: value tensor of shape [299, 4096] cannot be broadcast to indexing result of shape [598, 4096] Process EngineCore_0: Traceback (most recent call last): File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 636, in run_engine_core raise e File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core engine_core.run_busy_loop() File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop self._process_engine_step() File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step outputs, model_executed = self.step_fn() ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 267, in step model_output = self.execute_model_with_error_logging( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 253, in execute_model_with_error_logging raise err File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 244, in execute_model_with_error_logging return model_fn(scheduler_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model output = self.collective_rpc("execute_model", ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc answer = run_method(self.driver_worker, method, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/utils/init.py", line 2985, in run_method return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 337, in execute_model output = self.model_runner.execute_model(scheduler_output, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1393, in execute_model self._execute_mm_encoder(scheduler_output) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1103, in _execute_mm_encoder self.encoder_cache[req_id][input_id] = scatter_mm_placeholders( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/utils.py", line 67, in scatter_mm_placeholders placeholders[is_embed] = embeds ~~~~~~~~~~~~^^^^^^^^^^ RuntimeError: shape mismatch: value tensor of shape [299, 4096] cannot be broadcast to indexing result of shape [598, 4096]

I also happen to be short of exactly twice this amount.

[DarkLight1337]

Also cc @Isotr0py