Release - SuperBench v0.12.0 (#729)

polarG · Hongtao Zhang · web-flow · commit 0b4311cd7d54 · 2025-08-12T10:18:33.000-07:00
**Description** Cherry-pick bug fixes from v0.12.0 to main. **Major Revisions** * #725 * #727 * #728 Co-authored-by: Hongtao Zhang <hongtaozhang@microsoft.com> Co-authored-by: Yifan Xiong <yixio@microsoft.com> Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com> --------- Co-authored-by: Hongtao Zhang <hongtaozhang@microsoft.com>
diff --git a/.github/workflows/build-image.yml b/.github/workflows/build-image.yml
@@ -185,6 +185,11 @@ jobs:
           sources: >-
             superbench/main:cuda12.8-amd64
             superbench/main:cuda12.8-arm64
+        - name: cuda12.9
+          tags: superbench/main:cuda12.9
+          sources: >-
+            superbench/main:cuda12.9-amd64
+            superbench/main:cuda12.9-arm64
     steps:
     - name: Checkout
       uses: actions/checkout@v4
diff --git a/README.md b/README.md
@@ -15,7 +15,7 @@
 
 __SuperBench__ is a validation and profiling tool for AI infrastructure.
 
-📢 [v0.11.0](https://github.com/microsoft/superbenchmark/releases/tag/v0.11.0) has been released!
+📢 [v0.12.0](https://github.com/microsoft/superbenchmark/releases/tag/v0.12.0) has been released!
 
 ## _Check [aka.ms/superbench](https://aka.ms/superbench) for more details._
 
diff --git a/docs/getting-started/installation.mdx b/docs/getting-started/installation.mdx
@@ -61,7 +61,7 @@ You can clone the source from GitHub and build it.
 :::note Note
 You should checkout corresponding tag to use release version, for example,
 
-`git clone -b v0.11.0 https://github.com/microsoft/superbenchmark`
+`git clone -b v0.12.0 https://github.com/microsoft/superbenchmark`
 :::
 
 ```bash
diff --git a/docs/getting-started/run-superbench.md b/docs/getting-started/run-superbench.md
@@ -27,7 +27,7 @@ sb deploy -f remote.ini --host-password [password]
 :::note Note
 You should deploy corresponding Docker image to use release version, for example,
 
-`sb deploy -f local.ini -i superbench/superbench:v0.11.0-cuda12.4`
+`sb deploy -f local.ini -i superbench/superbench:v0.12.0-cuda12.9`
 
 You should note that version of git repo only determines version of sb CLI, and not the sb container. You should define the container version even if you specified a release version for the git clone.
 
diff --git a/docs/superbench-config.mdx b/docs/superbench-config.mdx
@@ -70,7 +70,7 @@ superbench:
 <TabItem value='example'>
 
 ```yaml
-version: v0.11
+version: v0.12
 superbench:
   enable: benchmark_1
   monitor:
diff --git a/docs/user-tutorial/container-images.mdx b/docs/user-tutorial/container-images.mdx
@@ -30,6 +30,11 @@ available tags are listed below for all stable versions.
 
 | Tag                | Description                         |
 |--------------------|-------------------------------------|
+| v0.12.0-cuda12.9   | SuperBench v0.12.0 with CUDA 12.9   |
+| v0.12.0-cuda12.8   | SuperBench v0.12.0 with CUDA 12.8   |
+| v0.12.0-cuda12.4   | SuperBench v0.12.0 with CUDA 12.4   |
+| v0.12.0-cuda12.2   | SuperBench v0.12.0 with CUDA 12.2   |
+| v0.12.0-cuda11.1.1 | SuperBench v0.12.0 with CUDA 11.1.1 |
 | v0.11.0-cuda12.4   | SuperBench v0.11.0 with CUDA 12.4   |
 | v0.11.0-cuda12.2   | SuperBench v0.11.0 with CUDA 12.2   |
 | v0.11.0-cuda11.1.1 | SuperBench v0.11.0 with CUDA 11.1.1 |
diff --git a/docs/user-tutorial/data-diagnosis.md b/docs/user-tutorial/data-diagnosis.md
@@ -65,7 +65,7 @@ superbench:
 example:
 ```yaml
 # SuperBench rules
-version: v0.11
+version: v0.12
 superbench:
   rules:
     failure-rule:
diff --git a/docs/user-tutorial/result-summary.md b/docs/user-tutorial/result-summary.md
@@ -58,7 +58,7 @@ superbench:
 
 ```yaml title="Example"
 # SuperBench rules
-version: v0.11
+version: v0.12
 superbench:
   rules:
     kernel_launch:
diff --git a/superbench/__init__.py b/superbench/__init__.py
@@ -6,5 +6,5 @@
 Provide hardware and software benchmarks for AI systems.
 """
 
-__version__ = '0.11.0'
+__version__ = '0.12.0'
 __author__ = 'Microsoft'
diff --git a/superbench/benchmarks/model_benchmarks/megatron_gpt3.py b/superbench/benchmarks/model_benchmarks/megatron_gpt3.py
@@ -749,7 +749,7 @@ def _cal_params_count(self):
     'megatron-deepseek-v2', MegatronGPT, parameters='--model=deepseek', platform=Platform.ROCM
 )
 BenchmarkRegistry.register_benchmark(
-    'megatron-deepseek-v2',
+    'megatron-deepseek-v2-lite',
     MegatronGPT,
     parameters=(
         '--model=deepseek '
diff --git a/superbench/config/amd_mi100_hpe.yaml b/superbench/config/amd_mi100_hpe.yaml
@@ -3,7 +3,7 @@
 # Server:
 #   - Product: HPE Apollo 6500
 
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   var:
diff --git a/superbench/config/amd_mi100_z53.yaml b/superbench/config/amd_mi100_z53.yaml
@@ -4,7 +4,7 @@
 #   - Product: G482-Z53
 #   - Link: https://www.gigabyte.cn/FileUpload/Global/MicroSite/553/G482-Z53.html
 
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   var:
diff --git a/superbench/config/amd_mi300.yaml b/superbench/config/amd_mi300.yaml
@@ -1,5 +1,5 @@
 # SuperBench Config
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   var:
diff --git a/superbench/config/azure/inference/standard_nc64as_t4_v3.yaml b/superbench/config/azure/inference/standard_nc64as_t4_v3.yaml
@@ -1,4 +1,4 @@
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   monitor:
diff --git a/superbench/config/azure/inference/standard_nc96ads_a100_v4.yaml b/superbench/config/azure/inference/standard_nc96ads_a100_v4.yaml
@@ -1,4 +1,4 @@
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   monitor:
diff --git a/superbench/config/azure/inference/standard_nv18ads_a10_v5.yaml b/superbench/config/azure/inference/standard_nv18ads_a10_v5.yaml
@@ -1,4 +1,4 @@
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   monitor:
diff --git a/superbench/config/azure_ndmv4.yaml b/superbench/config/azure_ndmv4.yaml
@@ -3,7 +3,7 @@
 # Azure NDm A100 v4
 # reference: https://docs.microsoft.com/en-us/azure/virtual-machines/ndm-a100-v4-series
 
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   monitor:
diff --git a/superbench/config/azure_ndv4.yaml b/superbench/config/azure_ndv4.yaml
@@ -1,5 +1,5 @@
 # SuperBench Config
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   monitor:
diff --git a/superbench/config/azure_ndv5.yaml b/superbench/config/azure_ndv5.yaml
@@ -1,5 +1,5 @@
 # SuperBench Config
-version: v0.11
+version: v0.12
 superbench:
   enable:
   monitor:
diff --git a/superbench/config/default.yaml b/superbench/config/default.yaml
@@ -1,5 +1,5 @@
 # SuperBench Config
-version: v0.11
+version: v0.12
 superbench:
   enable: null
   monitor:
diff --git a/website/blog/2025-08-06-release-0-12.md b/website/blog/2025-08-06-release-0-12.md
@@ -0,0 +1,62 @@
+---
+slug: release-sb-v0.12
+title: Releasing SuperBench v0.12
+author: Guoshuai Zhao
+author_title: SuperBench Team
+author_url: https://github.com/guoshzhao
+author_image_url: https://github.com/guoshzhao.png
+tags: [superbench, announcement, release]
+---
+
+We are very happy to announce that **SuperBench 0.12.0 version** is officially released today!
+
+You can install and try SuperBench by following the [Getting Started Tutorial](https://microsoft.github.io/superbenchmark/docs/getting-started/installation).
+
+## SuperBench 0.12.0 Release Notes
+
+### SuperBench Improvements
+
+- Optimized cutlass build process for faster builds and smaller binaries.
+- Improve image build pipeline.
+- Add support for arm64 builds.
+- Upgrade pipeline dependencies.
+- Fix SuperBench installation and code lint issues.
+- Update Flake8 repository.
+- Add support for the latest Python versions.
+- Enhance error handling for `pkg_resources` imports.
+- Update ROCm image build labels.
+- Add CUDA 12.8 and CUDA 12.9 support.
+- Consolidate multi-architecture Docker images.
+- Upgrade runner OS to latest version.
+
+### Micro-benchmark Improvements
+
+- Add general CPU bandwidth and latency benchmarks.
+- Add nvbandwidth build process and benchmarks.
+- Add architecture support for 10.0 in gemm-flops.
+- Add GPU Stream micro benchmark.
+- Add FP4 GEMM FLOPS support in `cublaslt_gemm` benchmark.
+- Add Grace CPU support for CPU Stream benchmark.
+- Revise CPU Stream benchmark.
+- Fix NUMA error on Grace CPU in gpu-copy benchmark.
+- Bump onnxruntime-gpu dependency from 1.10.0 to 1.12.0.
+- Fix stderr message in gpu-copy benchmark.
+- Fix TensorRT inference parsing.
+- Handle N/A values in nvbandwidth benchmark.
+- Avoid unintended nvbandwidth function calls in all benchmarks.
+- Support CUDA arch flag and autotuning in `cublaslt` GEMM.
+
+## Model-benchmark Improvements
+
+- Add LLaMA-2 model benchmarks.
+- Add Mixture of Experts model benchmarks.
+- Add DeepSeek inference benchmark (AMD GPU).
+- Fix typos in documentation and code.
+
+### Result Analysis
+
+- Enhance logging for diagnosis rule baseline errors.
+
+### Documentation Updates
+
+- Update CODEOWNERS file.
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
@@ -101,7 +101,7 @@ module.exports = {
     announcementBar: {
       id: 'supportus',
       content:
-        '📢 <a href="https://microsoft.github.io/superbenchmark/blog/release-sb-v0.11">v0.11.0</a> has been released! ' +
+        '📢 <a href="https://microsoft.github.io/superbenchmark/blog/release-sb-v0.11">v0.12.0</a> has been released! ' +
         '⭐️ If you like SuperBench, give it a star on <a target="_blank" rel="noopener noreferrer" href="https://github.com/microsoft/superbenchmark">GitHub</a>! ⭐️',
     },
     algolia: {
diff --git a/website/package-lock.json b/website/package-lock.json
diff --git a/website/package.json b/website/package.json
@@ -1,6 +1,6 @@
 {
   "name": "superbench-website",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "private": true,
   "scripts": {
     "docusaurus": "docusaurus",

Original file line number	Diff line number	Diff line change
`@@ -749,7 +749,7 @@ def _cal_params_count(self):`
`749`	`749`	`'megatron-deepseek-v2', MegatronGPT, parameters='--model=deepseek', platform=Platform.ROCM`
`750`	`750`	`)`
`751`	`751`	`BenchmarkRegistry.register_benchmark(`
`752`		`- 'megatron-deepseek-v2',`
	`752`	`+ 'megatron-deepseek-v2-lite',`
`753`	`753`	`MegatronGPT,`
`754`	`754`	`parameters=(`
`755`	`755`	`'--model=deepseek '`