[Bug]: The maximum model length calculation should also take into account the kv-cache-dtype

[ErykCh]

🐛 Describe the bug

Using kv-cache-dtype fp8 gives twice as many tokens. However, the mechanism for calculating the maximum number of tokens does not take this into account and only allows the size as if the kv-cache-dtype flag were not set.

with kv-cache-dtype fp8 and --max-model-len 100000

Available KV cache memory: 10.55 GiB
GPU KV cache size: 230,528 tokens
[kv_cache_utils.py:868] Maximum concurrency for 100,000 tokens per request: 2.31x

with kv-cache-dtype fp8 and --max-model-len 221000

ValueError: To serve at least one request with the models's max seq len (221000), (10.12 GiB KV cache is needed, which is larger than the available KV cache memory (5.59 GiB). Based on the available memory, the estimated maximum model length is 122080. Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

It's strange that this flag changes the amount of available memory.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ErykCh]

with kv-cache-dtype fp8 and --max-model-len 100000

Model loading took 29.0962 GiB and 15.951048 seconds

Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json']

Free memory on device (43.91/44.4 GiB) on startup. Desired GPU memory utilization is (0.988, 43.87 GiB). Actual usage is 29.1 GiB for weight, 4.2 GiB for peak activation, 0.02 GiB for non-torch memory, and 0.61 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with --kv-cache-memory=10520345161 to fit into requested memory, or --kv-cache-memory=10567986688 to fully utilize gpu memory. Current kv cache memory in use is 11331942985 bytes.

Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 7204

with kv-cache-dtype fp8 and --max-model-len 222000

Model loading took 29.0962 GiB and 15.783990 seconds

Using default MoE config. Performance might be sub-optimal! Config file not found at ['/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128,128].json']

[gpu_worker.py:298] Available KV cache memory: 5.59 GiB