Environment
- GPU: Intel Data Center GPU Max 1100 (PVC)
- torch: 2.15.0.dev20260824+xpu (latest nightly, also reproduced on dev20260812)
- triton-xpu: 3.8.0+git1e2d42a0
- Python: 3.10.20
- OS: Linux x86_64
- Model package:
pip install omnivoice (k2-fsa/OmniVoice TTS)
Minimal Reproducer
import torch, numpy as np
from omnivoice import OmniVoice
REF_WAV = "harvard.wav" # any ~18s speech WAV
REF_TEXT = "The birch canoe slid on the smooth planks."
TEXT = "Hello."
# CPU: PASS
model_cpu = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="cpu", dtype=torch.float32)
audio_cpu = model_cpu.generate(text=TEXT, ref_audio=REF_WAV, ref_text=REF_TEXT)
assert len(audio_cpu[0]) > 0
print(f"CPU: PASS ({len(audio_cpu[0])} samples)")
del model_cpu
# XPU: FAIL — device-side assert terminates process
model_xpu = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="xpu:0", dtype=torch.float32)
audio_xpu = model_xpu.generate(text=TEXT, ref_audio=REF_WAV, ref_text=REF_TEXT)
print(f"XPU: PASS ({len(audio_xpu[0])} samples)")Reference audio can be any speech WAV file (~6-18 seconds).
Expected Behavior
Generation completes and produces valid audio samples (as it does on CPU).
Actual Behavior
Device-side assertion terminates process:
/__w/pytorch/pytorch/third_party/torch-xpu-ops/src/ATen/native/xpu/sycl/IndexKernelUtils.h:62:
operator(): global id: [...], local id: [...]
Assertion `ind >= 0 && ind < ind_dim_size_ && "vectorized gather kernel index out of bounds"` failed.
What We Ruled Out
| Test |
Result |
| CPU generation |
✅ Always passes |
| topk indices in bounds |
✅ Verified (all idx < input_numel) |
| Tensor dimensions match |
✅ Verified |
| Isolated topk + advanced indexing put (5000 iterations) |
✅ Does NOT reproduce |
Adding torch.xpu.synchronize() between ops |
❌ Does NOT prevent crash |
Replacing advanced indexing put with index_select() + scatter() at one call site |
❌ Does NOT fully fix |
| Different transformers version (5.12 vs 5.16) |
❌ Still crashes |
| Different GPU card |
❌ Still crashes |
| FP16 / BF16 / FP32 |
❌ All crash |
Analysis
The OOB gather originates from a SYCL kernel dispatch inside the model forward pass. The exact source is difficult to isolate because:
- The crash is intermittent (sometimes succeeds, sometimes asserts)
- Any Python-level instrumentation (hooks,
.cpu(), thread spawn) prevents reproduction by changing kernel scheduling
- The crash occurs inside
model.generate() which dispatches hundreds of kernels
We suspect a missing dependency barrier between dependent kernel launches in the SYCL command queue, causing one kernel to read uninitialized data from another.
Request
Help identifying which specific SYCL kernel dispatch triggers the OOB gather. Debug tracing of IndexKernelUtils.h:62 call sites would help narrow down the root cause.
Environment
pip install omnivoice(k2-fsa/OmniVoice TTS)Minimal Reproducer
Reference audio can be any speech WAV file (~6-18 seconds).
Expected Behavior
Generation completes and produces valid audio samples (as it does on CPU).
Actual Behavior
Device-side assertion terminates process:
What We Ruled Out
torch.xpu.synchronize()between opsindex_select()+scatter()at one call siteAnalysis
The OOB gather originates from a SYCL kernel dispatch inside the model forward pass. The exact source is difficult to isolate because:
.cpu(), thread spawn) prevents reproduction by changing kernel schedulingmodel.generate()which dispatches hundreds of kernelsWe suspect a missing dependency barrier between dependent kernel launches in the SYCL command queue, causing one kernel to read uninitialized data from another.
Request
Help identifying which specific SYCL kernel dispatch triggers the OOB gather. Debug tracing of
IndexKernelUtils.h:62call sites would help narrow down the root cause.