Skip to content

CI: add top-priority models to vLLM benchmark#2952

Open
gyohuangxin wants to merge 5 commits intomainfrom
ci/vllm-benchmark-top-priority-models
Open

CI: add top-priority models to vLLM benchmark#2952
gyohuangxin wants to merge 5 commits intomainfrom
ci/vllm-benchmark-top-priority-models

Conversation

@gyohuangxin
Copy link
Copy Markdown
Member

Refs #2826

Summary

  • replace the generic vLLM benchmark matrix with explicit top-priority lanes for openai/gpt-oss-120b, deepseek-ai/DeepSeek-R1-0528, and moonshotai/Kimi-K2.5
  • keep the benchmark execution flow close to the existing workflow while allowing model-specific TP, extra args, and Kimi-only env overrides
  • align main-branch vLLM CI coverage with the current top-priority benchmark list from issue Increase vLLM downstream test coverage #2826

Test plan

  • parse .github/workflows/vllm_benchmark.yaml with python3 and yaml.safe_load
  • verify the isolated workflow diff only changes vllm_benchmark.yaml
  • run the GitHub Actions workflow end-to-end

Replace the old vLLM benchmark matrix with explicit GPT-OSS, DeepSeek-R1-0528, and Kimi K2.5 lanes so main-branch CI reflects the top-priority model coverage tracked in issue #2826.
@gyohuangxin gyohuangxin requested review from a team and Copilot April 29, 2026 06:49
@github-actions
Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests
ci:atom ATOM benchmark (DeepSeek-R1 + GPT-OSS)
ci:vllm vLLM benchmark
ci:all All of the above

Add labels via the sidebar or gh pr edit 2952 --add-label <label>

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the vLLM downstream benchmark CI workflow to run a small, explicit set of “top-priority” model lanes with per-model configuration (TP sizing, extra CLI args, and optional env overrides) instead of the previous cartesian matrix.

Changes:

  • Replaces the model/kv-cache cartesian matrix with an explicit matrix.include list for three prioritized models.
  • Adds per-lane parameters (tp, extra_args, extra_env_args) and wires them into the benchmark invocation.
  • Keeps kv-cache dtype handling while letting each lane opt into fp8 KV cache and/or additional flags.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +133 to +136
kv_cache_dtype: default_kvcache
tp: 8
extra_args: --block-size 1 --trust-remote-code
extra_env_args: -e VLLM_USE_TRITON_FLASH_ATTN=0
Comment on lines +122 to +126
- model: openai/gpt-oss-120b
kv_cache_dtype: fp8_kvcache
tp: 1
extra_args: --gpu-memory-utilization 0.3
extra_env_args: ""
Remove the shared vLLM image build job and install the aiter checkout inside each benchmark lane so pull requests no longer need a separate image-build stage before running the top-priority model matrix.
Allow the vLLM benchmark workflow to rerun on pull request synchronize events so follow-up commits on labeled PRs automatically refresh the benchmark jobs.
Build the aiter wheel once on the wheel runner and have each vLLM benchmark lane install that artifact in the base image so benchmark jobs no longer depend on editable installs or per-lane source rebuilds.
Switch the GPT-OSS lane to tensor parallel size 8 so the vLLM benchmark no longer tries to reserve KV cache on a single GPU budget that is too small for the model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants