[benchmarks][ci] Initial integration of sglang kernels to benchmarks #3796

LiyangLingIntel · 2025-03-31T05:55:41Z

The initial enabling for sglang benchmarks.
Include sglang prefill/decode/extended attention and fp8 quant gemm into third-party benchmark.

.github/workflows/third-party-benchmarks.yml

benchmarks/third_party/sglang/sglang.patch

.github/workflows/third-party-tests.yml

.github/workflows/third-party-benchmarks.yml

.github/pins/sglang.txt

.github/workflows/third-party-benchmarks.yml

LiyangLingIntel · 2025-04-15T05:27:21Z

Benchmark is still blocked by #3748, #3749.
Let's merge this PR when the blocking issues are resolved with the new agama release, expect in late April.

etiotto · 2025-04-24T12:22:28Z

@LiyangLingIntel the test are passing. Is this PR still going to wait on #3748 and #3749 ?

LiyangLingIntel · 2025-04-24T14:21:44Z

@LiyangLingIntel the test are passing. Is this PR still going to wait on #3748 and #3749 ?

@etiotto Yes, these 2 depend on the new agama release.
The target workflow is Triton Third-party benchmark, it is not included in CI and scheduled once per day.

LiyangLingIntel · 2025-05-22T09:08:26Z

Verified the blocking issues #3748, #3749 are fixed on rolling agama 2507.

benchmarks/triton_kernels_benchmark/build_report.py

LiyangLingIntel · 2025-05-26T01:53:00Z

The third party benchmark passed here https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/15210819443/job/42784278091

whitneywhtsang · 2025-05-27T17:27:43Z

@Egor-Krivov @vlad-penkin Please take another look.

.github/workflows/third-party-benchmarks.yml

vlad-penkin · 2025-05-28T19:16:14Z

benchmarks/third_party/sglang/block_fp8_gemm_benchmark.py

Let's move all sglang benchmarks to the benchmarks/triton_kernels_benchmark subfolder so it can be part of the package.

I would not suggest to do so.

Currently we do not have performance goal on sglang. There is no need to integrate these kernels to triton_kernels_benchmark, it's better to make it focus on high priority kernels. It can help save our CI time.

SGLang is changing frequently and we do not pin to any stable commits of sglang. Moving these sglang benchmark to triton_kernels_benchmark would make it fragile.

I wonder if you can specify our strategy on these third party kernels benchmark systematically, including sglang, liger-kernels and other potential third parties(like FlagGems).
Which kernels we want to trace the peak performance and which we just want to know what performance it can be, or just ensure there is no regression?
Should all third parties be in the same strategy, or there are priorities on these third parties?

Currently we do not have performance goal on sglang. There is no need to integrate these kernels to triton_kernels_benchmark, it's better to make it focus on high priority kernels. It can help save our CI time.

AFAIK we have experimental category just for this. We can also disable their launch in CI regardless of the location of the benchmark, right?

SGLang is changing frequently and we do not pin to any stable commits of sglang. Moving these sglang benchmark to triton_kernels_benchmark would make it fragile.

We have, I believe, a well-established procedure for working with third-party sources (like PyTorch, spirv-llvm translator). We use a commit pin. If we need to periodically check the pin on the main, we can set up a workflow for this.

vlad-penkin · 2025-05-28T19:16:32Z

benchmarks/third_party/sglang/decode_attention_benchmark.py

Let's move all sglang benchmarks to the benchmarks/triton_kernels_benchmark subfolder so it can be part of the package.

Let's also integrate benchmarks into the package. The sample integration is

https://github.com/intel/intel-xpu-backend-for-triton/blob/main/benchmarks/triton_kernels_benchmark/gemm_benchmark.py

intel-xpu-backend-for-triton/benchmarks/triton_kernels_benchmark/configs/benchmark_config_templates.py

Line 21 in 33f5599

BenchmarkConfig(

vlad-penkin · 2025-05-28T19:16:46Z

benchmarks/third_party/sglang/extended_attention_benchmark.py

Let's move all sglang benchmarks to the benchmarks/triton_kernels_benchmark subfolder so it can be part of the package.

Let's also integrate benchmarks into the package. The sample integration is

https://github.com/intel/intel-xpu-backend-for-triton/blob/main/benchmarks/triton_kernels_benchmark/gemm_benchmark.py

intel-xpu-backend-for-triton/benchmarks/triton_kernels_benchmark/configs/benchmark_config_templates.py

Line 21 in 33f5599

BenchmarkConfig(

vlad-penkin · 2025-05-28T19:17:19Z

benchmarks/third_party/sglang/prefill_attention_benchmark.py

Let's move all sglang benchmarks to the benchmarks/triton_kernels_benchmark subfolder so it can be part of the package.

Let's also integrate benchmarks into the package. The sample integration is

https://github.com/intel/intel-xpu-backend-for-triton/blob/main/benchmarks/triton_kernels_benchmark/gemm_benchmark.py

intel-xpu-backend-for-triton/benchmarks/triton_kernels_benchmark/configs/benchmark_config_templates.py

Line 21 in 33f5599

BenchmarkConfig(

benchmarks/third_party/sglang/decode_attention_benchmark.py

benchmarks/third_party/sglang/extended_attention_benchmark.py

benchmarks/third_party/sglang/prefill_attention_benchmark.py

vlad-penkin · 2025-05-28T19:31:43Z

.github/workflows/third-party-benchmarks.yml


      - name: Install benchmark dependencies
        id: install
        run: |
          pip install transformers pandas pytest

+          cd benchmarks
+          pip install .
+          pip install intel-pti==0.12.2


Why this line is needed?

Cause sglang benchmark has very few to do with Triton, we want leverage our benchmarking tools to do profiling. These lines are same to

intel-xpu-backend-for-triton/.github/workflows/triton-benchmarks.yml

Line 125 in 91066ce

cd benchmarks

.

Port prefill attn and decode attn from sglang Add validation temp add extend attention disable debug ir dump Update three stage attention benchmark Add sglang kernel benchmark to action use 1e-3 atol remove sglang benchmark from triton-benchmarks Fix setup bdist_wheel Add sglang to thirdparty test Address review comments Remove sglang from tests Fix CI Address review comments Integrate sglang prefill/decode/extend kernel to benchmarks Port prefill attn and decode attn from sglang Add validation temp add extend attention disable debug ir dump Update three stage attention benchmark Add sglang kernel benchmark to action use 1e-3 atol remove sglang benchmark from triton-benchmarks Fix setup bdist_wheel Add sglang to thirdparty test Address review comments Remove sglang from tests Adjust params term Adjust tflops computation

fix bugs rtol atol Move fp8 gemm to sglang benchmark

Address review comments Fix CI XPU not found

LiyangLingIntel self-assigned this Mar 31, 2025

LiyangLingIntel linked an issue Mar 31, 2025 that may be closed by this pull request

[SGLANG] Initial enabling and its integration into the test scripts #3623

Closed

LiyangLingIntel requested a review from vlad-penkin March 31, 2025 05:56

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch 3 times, most recently from b43b5e2 to bbd10a8 Compare April 9, 2025 02:34

LiyangLingIntel linked an issue Apr 9, 2025 that may be closed by this pull request

Enhance FlashAttention and FlexAttention benchmarks to evaluate Decode and Append stages #3617

Closed

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch 2 times, most recently from 34e464a to 29e2711 Compare April 10, 2025 06:07

LiyangLingIntel marked this pull request as ready for review April 10, 2025 07:47

LiyangLingIntel requested review from whitneywhtsang and Egor-Krivov April 10, 2025 07:48

Egor-Krivov requested changes Apr 10, 2025

View reviewed changes

.github/workflows/third-party-benchmarks.yml Outdated Show resolved Hide resolved

.github/workflows/third-party-benchmarks.yml Outdated Show resolved Hide resolved

.github/workflows/third-party-benchmarks.yml Outdated Show resolved Hide resolved

vlad-penkin requested changes Apr 10, 2025

View reviewed changes

vlad-penkin requested review from anmyachev and pbchekin April 10, 2025 13:45

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch from 29e2711 to 4b7241c Compare April 11, 2025 05:33

LiyangLingIntel mentioned this pull request Apr 11, 2025

[SGLANG] add sglang block fp8 gemm kernels into benchmark #3676

Open

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch 2 times, most recently from 1296327 to 7d4d837 Compare April 14, 2025 08:15

LiyangLingIntel requested review from vlad-penkin and Egor-Krivov April 14, 2025 12:28

Egor-Krivov requested changes Apr 14, 2025

View reviewed changes

.github/workflows/third-party-benchmarks.yml Outdated Show resolved Hide resolved

whitneywhtsang requested a review from Egor-Krivov April 15, 2025 07:59

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch from 9651ebe to e609f5d Compare April 15, 2025 08:14

etiotto marked this pull request as draft April 17, 2025 14:27

etiotto added the dependencies: igc label Apr 28, 2025

LiyangLingIntel linked an issue May 15, 2025 that may be closed by this pull request

[SGLANG] add sglang block fp8 gemm kernels into benchmark #3676

Open

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch from e609f5d to 3853afa Compare May 22, 2025 08:21

LiyangLingIntel marked this pull request as ready for review May 22, 2025 08:32

Egor-Krivov requested changes May 22, 2025

View reviewed changes

benchmarks/triton_kernels_benchmark/build_report.py Outdated Show resolved Hide resolved

whitneywhtsang requested a review from Egor-Krivov May 22, 2025 22:54

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch 5 times, most recently from 48b96ec to 458e06d Compare May 23, 2025 12:53

Egor-Krivov approved these changes May 28, 2025

View reviewed changes

vlad-penkin requested changes May 28, 2025

View reviewed changes

LiyangLingIntel and others added 3 commits May 29, 2025 01:14

add sglang block fp8 gemm into benchmark

4f1195f

fix bugs rtol atol Move fp8 gemm to sglang benchmark

Update extended attention interface

211e82d

Address review comments Fix CI XPU not found

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch from 458e06d to 10da795 Compare May 29, 2025 02:33

Address review comments

ead512a

LiyangLingIntel force-pushed the liyang/init_sglang_benchmark branch from 10da795 to ead512a Compare May 29, 2025 07:18

Merge branch 'main' into liyang/init_sglang_benchmark

0ecf2a6

[benchmarks][ci] Initial integration of sglang kernels to benchmarks #3796

Are you sure you want to change the base?

[benchmarks][ci] Initial integration of sglang kernels to benchmarks #3796

Conversation

LiyangLingIntel commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LiyangLingIntel commented Apr 15, 2025

Uh oh!

etiotto commented Apr 24, 2025

Uh oh!

LiyangLingIntel commented Apr 24, 2025

Uh oh!

LiyangLingIntel commented May 22, 2025

Uh oh!

Uh oh!

LiyangLingIntel commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whitneywhtsang commented May 27, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anmyachev May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LiyangLingIntel commented Mar 31, 2025 •

edited

Loading

LiyangLingIntel commented May 26, 2025 •

edited

Loading

anmyachev May 30, 2025 •

edited

Loading