Add aiter qknorm and rope fusion kernel

[kliuae-amd]

Purpose

TL;DR: To enable qknorm and rope fusion, set VLLM_ROCM_USE_AITER_TRITON_ROPE=0, and launch vLLM with --compilation-config '{"pass_config": {"enable_qk_norm_rope_fusion": "true"}}'.

This PR adds qknorm and rope fusion kernel from aiter. This can be controlled through VLLM_ROCM_USE_AITER_FUSED_QK_NORM_ROPE.

This flag is defaulted to false for now, however, as vLLM has already had a native CUDA kernel for qknorm and rope fusion, and on MI300X both end-to-end and kernel test show that the performance of this aiter kernel is similar to, and even slightly worse than the vLLM counterpart.

To enable the fusion, the following compilation config needs to be set: --compilation-config '{"pass_config": {"enable_qk_norm_rope_fusion": "true"}}'.

For this fusion to work, however, aiter's triton rope needs to be disabled as the compiled triton kernel cannot be found by the pattern matcher. This PR also disables the VLLM_ROCM_USE_AITER_TRITON_ROPE by default for aggressive ops fusion.

Test Plan

Qwen/Qwen3-30B-A3B-Instruct-2507

server

VLLM_ROCM_USE_AITER_FUSED_QK_NORM_ROPE=1 \
VLLM_ROCM_USE_AITER_TRITON_ROPE=0 \
VLLM_USE_V1=1 \
SAFETENSORS_FAST_GPU=1 \
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MOE=1 \
VLLM_ROCM_USE_AITER_MHA=1 \
vllm serve $MODEL \
 -tp 1 \
 --gpu-memory-utilization 0.8 \
 --trust_remote_code \
 --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "pass_config": {"enable_qk_norm_rope_fusion": "true"}}' \
 --no-enable-prefix-caching

benchmark

vllm bench serve --model $MODEL --dataset-name random --random-input-len 1000 --random-output-len 1000 --num-warmups 10 --num-prompts 1000 --max-concurrency 64

lm_eval

lm_eval --model local-completions --tasks gsm8k --model_args model=$MODEL,base_url=http://127.0.0.1:8000/v1/completions --batch_size 100

Test Result

Using old default w/o fusion

VLLM_ROCM_USE_AITER_TRITON_ROPE=1
VLLM_ROCM_USE_AITER_FUSED_QK_NORM_ROPE=0
--compilation-config '{"pass_config": {"enable_qk_norm_rope_fusion": "false"}}'

============= Serving Benchmark Result =============
Successful requests:                     1000
Failed requests:                         0
Maximum request concurrency:             64
Benchmark duration (s):                  336.18
Total input tokens:                      1000000
Total generated tokens:                  1000000
Request throughput (req/s):              2.97
Output token throughput (tok/s):         2974.57
Peak output token throughput (tok/s):    3392.00
Peak concurrent requests:                128.00
Total Token throughput (tok/s):          5949.13
----------------Time to First Token-----------------
Mean TTFT (ms):                          458.15
Median TTFT (ms):                        482.17
P99 TTFT (ms):                           1433.09
------Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.70
Median TPOT (ms):                        20.86
P99 TPOT (ms):                           21.95
-----------------Inter-token Latency----------------
Mean ITL (ms):                           20.70
Median ITL (ms):                         20.07
P99 ITL (ms):                            21.97
====================================================

Fusion with vLLM's fusion kernel (new default)

VLLM_ROCM_USE_AITER_TRITON_ROPE=0
VLLM_ROCM_USE_AITER_FUSED_QK_NORM_ROPE=0
--compilation-config '{"pass_config": {"enable_qk_norm_rope_fusion": "true"}}'

============= Serving Benchmark Result =============
Successful requests:                     1000
Failed requests:                         0
Maximum request concurrency:             64
Benchmark duration (s):                  320.32
Total input tokens:                      1000000
Total generated tokens:                  1000000
Request throughput (req/s):              3.12
Output token throughput (tok/s):         3121.92
Peak output token throughput (tok/s):    3584.00
Peak concurrent requests:                128.00
Total Token throughput (tok/s):          6243.83
----------------Time to First Token-----------------
Mean TTFT (ms):                          439.01
Median TTFT (ms):                        463.04
P99 TTFT (ms):                           1379.98
------Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          19.71
Median TPOT (ms):                        19.67
P99 TPOT (ms):                           21.56
-----------------Inter-token Latency----------------
Mean ITL (ms):                           19.71
Median ITL (ms):                         18.95
P99 ITL (ms):                            21.42
====================================================

Fusion with aiter's fusion kernel

VLLM_ROCM_USE_AITER_TRITON_ROPE=0
VLLM_ROCM_USE_AITER_FUSED_QK_NORM_ROPE=1
--compilation-config '{"pass_config": {"enable_qk_norm_rope_fusion": "true"}}'

============= Serving Benchmark Result =============
Successful requests:                     1000
Failed requests:                         0
Maximum request concurrency:             64
Benchmark duration (s):                  322.36
Total input tokens:                      1000000
Total generated tokens:                  1000000
Request throughput (req/s):              3.10
Output token throughput (tok/s):         3102.16
Peak output token throughput (tok/s):    3520.00
Peak concurrent requests:                128.00
Total Token throughput (tok/s):          6204.32
----------------Time to First Token-----------------
Mean TTFT (ms):                          443.45
Median TTFT (ms):                        470.12
P99 TTFT (ms):                           1379.35
------Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          19.85
Median TPOT (ms):                        19.96
P99 TPOT (ms):                           21.87
-----------------Inter-token Latency----------------
Mean ITL (ms):                           19.85
Median ITL (ms):                         19.23
P99 ITL (ms):                            21.37
====================================================

lm_eval w/ aiter fusion kernel

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|_  |0.8870|_  |0.0087|
|     |       |strict-match    |     5|exact_match|_  |0.8779|_  |0.0090|

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.