Summary
On BMG (Arc B580, Linux), the AnnotateCacheControl pass re-enabled by #7717 costs 1.58x end-to-end on TorchBench pyhpc_isoneutral_mixing — but only on the plain-load path, i.e. when TRITON_INTEL_PREDICATED_LOAD=0. On the predicated-load path (today's default) the pass is neutral, so the regression is currently hidden.
This matters for #7755: one of the options under discussion there is reverting the predicated-load default to off. Doing that would expose this regression, which is larger (1.58x) than the predicated-load one it would be working around (1.25x).
This looks like a Linux/BMG instance of the same class of regression that #7717 already keeps the pass gated off for on Windows (#7495).
Measurements
fp16 inference, clean main @ b1aaf9df7, 3 reps per cell, spread <= 0.3%:
TRITON_INTEL_PREDICATED_LOAD |
AnnotateCacheControl |
latency |
| 0 |
on (default) |
5.88 ms |
| 0 |
off |
3.72 ms |
| 1 (default) |
on (default) |
4.63 ms |
| 1 |
off |
4.64 ms |
Reading down the first two rows: 1.58x on the plain-load path. Reading down the last two: 0.3% (noise) on the predicated path.
Reproducer
python benchmarks/dynamo/torchbench.py -d xpu --performance --inference \
--backend=inductor --cold-start-latency -n10 --disable-cudagraphs \
--float16 --only pyhpc_isoneutral_mixing
A/B with, per arm:
TRITON_INTEL_PREDICATED_LOAD=0 # regressed
TRITON_INTEL_PREDICATED_LOAD=0 TRITON_INTEL_DISABLE_ANNOTATE_CACHE_CONTROL=1 # fast
Note the inverted sense of the second variable (1 = pass off). Both variables are latched in function-local statics, so each arm needs its own process, and each arm needs its own TORCHINDUCTOR_CACHE_DIR — Triton env vars are not part of Inductor's autotune cache key, so a shared cache dir makes the flag look inert.
What the pass actually emits
Comparing the generated LLVM IR for the whole 56-kernel wrapper, pass on vs off:
- Plain-load path — 1033 of 5117
load instructions gain !nontemporal. That is the entire difference; load/store vectorization widths and branch counts are identical.
- Predicated-load path — cache control is carried as
!spirv.DecorationCacheControlINTEL metadata instead (1380 additional decoration lines). The pass is not inert here; the decorations simply cost nothing.
So the suspicion is that the !nontemporal hints on ordinary loads are the expensive part on this kernel — these are stencil-style pointwise kernels with heavy neighbour reuse, where marking loads non-temporal defeats the cache reuse the kernel depends on.
Environment
- GPU: Intel Arc B580 (BMG)
main @ b1aaf9df7- ocloc 26.31.39395.13, Level Zero driver 1.17.39395+13
- TorchBench
pyhpc_isoneutral_mixing, fp16, inference, Inductor
Suggested next step
Confirm whether the !nontemporal annotation is intended for loads with expected reuse, and if not, restrict where the pass applies it. A blunter alternative — extending the Windows gate to BMG — would also fix this, but would give up whatever the pass was enabled for in #7579.
Related
Summary
On BMG (Arc B580, Linux), the
AnnotateCacheControlpass re-enabled by #7717 costs 1.58x end-to-end on TorchBenchpyhpc_isoneutral_mixing— but only on the plain-load path, i.e. whenTRITON_INTEL_PREDICATED_LOAD=0. On the predicated-load path (today's default) the pass is neutral, so the regression is currently hidden.This matters for #7755: one of the options under discussion there is reverting the predicated-load default to off. Doing that would expose this regression, which is larger (1.58x) than the predicated-load one it would be working around (1.25x).
This looks like a Linux/BMG instance of the same class of regression that #7717 already keeps the pass gated off for on Windows (#7495).
Measurements
fp16 inference, clean
main@b1aaf9df7, 3 reps per cell, spread <= 0.3%:TRITON_INTEL_PREDICATED_LOADAnnotateCacheControlReading down the first two rows: 1.58x on the plain-load path. Reading down the last two: 0.3% (noise) on the predicated path.
Reproducer
A/B with, per arm:
Note the inverted sense of the second variable (
1= pass off). Both variables are latched in function-local statics, so each arm needs its own process, and each arm needs its ownTORCHINDUCTOR_CACHE_DIR— Triton env vars are not part of Inductor's autotune cache key, so a shared cache dir makes the flag look inert.What the pass actually emits
Comparing the generated LLVM IR for the whole 56-kernel wrapper, pass on vs off:
loadinstructions gain!nontemporal. That is the entire difference; load/store vectorization widths and branch counts are identical.!spirv.DecorationCacheControlINTELmetadata instead (1380 additional decoration lines). The pass is not inert here; the decorations simply cost nothing.So the suspicion is that the
!nontemporalhints on ordinary loads are the expensive part on this kernel — these are stencil-style pointwise kernels with heavy neighbour reuse, where marking loads non-temporal defeats the cache reuse the kernel depends on.Environment
main@b1aaf9df7pyhpc_isoneutral_mixing, fp16, inference, InductorSuggested next step
Confirm whether the
!nontemporalannotation is intended for loads with expected reuse, and if not, restrict where the pass applies it. A blunter alternative — extending the Windows gate to BMG — would also fix this, but would give up whatever the pass was enabled for in #7579.Related
AnnotateCacheControlpass by default back #7579 — the request to re-enable itAnnotateCacheControlpass by default to avoid perf regressions in E2E (#7566) #7578 / [TritonIntelGPU] DisableAnnotateCacheControlpass by default to avoid perf regressions in E2E #7566 — the earlier revert/disable for the same reason