Skip to content

CUTracer v0.3.0 Release Notes

Latest
Latest

Choose a tag to compare

View all tags
@FindHao FindHao released this 10 Jun 03:38
v0.3.0
dde45fd

🎉 Minor Release — 49 commits since v0.2.1

CUTracer v0.3.0 (2026-04-23 → 2026-06-09) is a substantial minor release that focuses on three big themes: (1) a new high-performance RapidJSON trace serializer that becomes the default and removes ~16% of host-CPU overhead, (2) the completed NVBit 1.8 first-class TMA capture pipeline, which records fully-structured tensor metadata instead of opaque parameter handles, and (3) warpgroup-targeted random-delay injection that can stall an entire warp or warpgroup as a single scheduling unit to reproduce cross-warpgroup scheduler races. The release also expands the trace schemas to cover every record type the writer emits (with a CI drift guard), and migrates the repository from facebookresearch to facebookexperimental with the wiki now living under docs/.

✨ Highlights

  • RapidJSON trace serializer (default) — Removes ~16% host CPU and ~8% malloc churn from the per-record write path; byte-identical output to the legacy nlohmann path for reg/mem/opcode/mem_value records.
  • Warpgroup-targeted random delay — --delay-warpgroup-id / --delay-warp-mask / --delay-enable-prob flags let random_delay stall an entire warp or warpgroup as a single scheduling unit, enabling reproduction of cross-warpgroup scheduler races that per-thread delays could not.
  • NVBit 1.8 TMA capture pipeline — Migration to first-class nvbit_parse_tma_transfer_info(); the trace now carries structured tensor metadata (dim, dtype, strides, swizzle, SMEM dst/src/mbar addresses) instead of opaque parameter handles.
  • Complete trace-record schemas + drift guard — New JSON Schemas for tma_trace, mem_value_trace, and mem_addr_trace, plus a CI guard that catches writer/schema divergence.
  • Repository migrated to facebookexperimental — Wiki content moved into docs/ with an auto-sync workflow; CI gains a self-hosted H100 lane and retires the legacy T4 GitHub-hosted workflow.

🏎️ Capture-Side Serializer Overhaul (RapidJSON)

A capture CPU profile attributed ~16% of host CPU to per-record nlohmann::json DOM serialization plus ~8% malloc churn from per-record std::stringstream hex formatting. This release ports the entire hot path to a streaming rapidjson::Writer and makes it the default.

  • Opt-in RapidJSON path — Adds a streaming Writer into a reused thread_local StringBuffer for reg_trace / mem_addr_trace / mem_value_trace / opcode_only; during the migration window this was gated behind CUTRACER_JSON_ENGINE={nlohmann,rapidjson,ab} with an ab oracle that semantically compared both engines per record and reported per-type match/mismatch counters at writer teardown.
  • Unit-test the RapidJSON serializer — Extracted to its own translation unit with cxx goldens, exercising all four record types.
  • Port tma_trace — Removes the last nlohmann fallback in the per-record path; semantically identical on 2,304 TMA descriptors.
  • RapidJSON becomes the default — Removes the CUTRACER_JSON_ENGINE switch, the A/B comparison machinery, build_nlohmann_line, and the entire per-record nlohmann serializer family (~390 lines net). nlohmann is retained only for the one-time kernel_metadata header.
  • Heap-corruption fix in TraceWriter (#225) — Serialize public mutators with std::mutex to prevent libstdc++ _M_mutate heap double-free that surfaced as "JSON decode error - unexpected character" at ~7% per kernel-run on aarch64. Replaces a misleading "CRITICAL FIX" std::move dance that did not actually fix the race.
  • OSS install fix (#224) — install_third_party.sh now downloads rapidjson headers so the OSS CI build succeeds.

🎯 Warpgroup-Targeted Random Delay

Per-thread random delay washes out at the warp boundary (warps are lock-step, so the effective stall is max(thread_delays)), and existing CTA-local modes operate on clusters, not warp ranges within a single CTA. This release teaches random_delay to stall an entire warp or warpgroup as a single scheduling unit — the exact timing distribution needed to reproduce cross-warpgroup scheduler races such as SM100 flash-attention TMEM dealloc.

  • Device function + host wrapper — New instrument_delay_warpgroup(pred, delay_ns, warp_mask) in inject_funcs.cu and instrument_warpgroup_delay_injection host wrapper. Computes CTA-local warp id from threadIdx/blockDim, tests the warp_mask bit, and __nanosleeps the entire warp uniformly via the same 1ms chunked loop as instrument_delay_random_cluster.
  • CLI flags + env vars — --delay-warp-mask (CUTRACER_DELAY_WARP_MASK; hex or decimal CTA-local warp bitmask) and --delay-warpgroup-id (CUTRACER_DELAY_WARPGROUP_ID; integer warpgroup index that resolves to 0xF << (id * 4) on the host and wins over --delay-warp-mask when both are set).
  • Dispatcher routing + --delay-enable-prob — Routes warp targeting through the existing delay dispatcher in cutracer.cu and adds a probability gate (default 0.5 preserves the historical 50/50 PC gate; set to 1.0 for deterministic injection — strongly recommended with warp targeting).
  • Persist warp targeting in delay config JSON — Replay mode (--delay-load-path) now round-trips warp/warpgroup targeting for deterministic reproduction.

📡 TMA Pipeline (NVBit 1.8 Migration)

Migrates TMA tracing off non-public ISA semantics (URa/URa+1 layout, MULTICAST register-shift workarounds) and onto NVBit 1.8's first-class TMA APIs.

  • GPU-side capture — Asks NVBit for the TMA parameter handle directly via the new public API.
  • Host-side parsing — Calls nvbit_parse_tma_transfer_info() from the receive thread to decode the runtime handle into a structured TMATransferInfo_t; the trace now carries fully-structured tensor metadata (dim, mode, coords, dtype, rank, global address/dim/strides, box dim, swizzle).
  • SMEM dst/src addresses — Serializes dst.data_address, dst.mbar_address (and src.* for UTMASTG) from NVBit 1.8's TMAAddress_t union. These are the official NVBit API equivalent of the URb/URb+1 register values, so capture no longer relies on implicit register snapshotting in reg_trace.

🧪 Schema & Validation

  • Trace schemas for every record type + drift guard — Adds JSON Schemas for tma_trace, mem_value_trace, and mem_addr_trace. Previously SCHEMAS_BY_TYPE only covered reg_trace / mem_trace / opcode_only / kernel_metadata, so validate / compare silently skipped every current trace's data records. A new CI drift guard catches writer/schema divergence.
  • SM100 uniform-register indices — reg_trace schema's uregs_indices maximum raised from 62 to 255 (matches regs_indices) to avoid chasing per-arch maximums; both are uint8_t register indices where high values encode special registers.

🖥️ CLI Changes

# New short alias for --output-dir (matches gcc/ffmpeg/curl/tar convention)
cutracer trace -o ./logs -- python my_kernel.py

# Warpgroup-targeted delay (stall warpgroup 1)
cutracer trace -i random_delay --delay-ns 100000 \
    --delay-warpgroup-id 1 \
    --delay-enable-prob 1.0 \
    -- python my_kernel.py

# Or an explicit CTA-local warp bitmask (warps 4-7)
cutracer trace -i random_delay --delay-ns 100000 \
    --delay-warp-mask 0xF0 \
    -- python my_kernel.py
  • Friendlier trace errors — Three UX fixes for the intentionally permissive trace subcommand (ignore_unknown_options=True, nargs=-1, type=click.UNPROCESSED): clearer messages when wrapped command tokens leak into CUTracer's own option parser, a warning when running under buck2 test, and absolute-path resolution for the wrapped command.
  • -o short alias — Closes the foot-gun where a stray -o ./logs was silently appended to the wrapped command (yielding /bin/sh: - : invalid option) because the trace subcommand passes unknown options through to the target binary.

📁 Configuration Changes

Removed Environment Variables

Variable Notes
CUTRACER_JSON_ENGINE The opt-in switch from the RapidJSON migration window. No longer required — RapidJSON is now the default and only per-record path; the nlohmann per-record code was removed.

New Environment Variables

Variable Description Default
CUTRACER_DELAY_WARP_MASK CTA-local warp bitmask for random_delay. Accepts hex (0xF) or decimal (15). 0 = disabled. 0
CUTRACER_DELAY_WARPGROUP_ID Warpgroup index; >= 0 selects warps [4N..4N+3] and wins over CUTRACER_DELAY_WARP_MASK. -1 = disabled. -1
CUTRACER_DELAY_ENABLE_PROB Probability per PC of enabling delay during recording (random mode). Set to 1.0 for deterministic injection. 0.5

🔄 Dependency Updates

  • NVBit: 1.7 → 1.8 (first-class TMA APIs: nvbit_parse_tma_transfer_info, TMATransferInfo_t, TMAAddress_t)
  • RapidJSON: now a vendored header dependency installed by install_third_party.sh (used as the default trace serializer)

🏗️ Infrastructure

  • Repository migration — facebookresearch/CUTracer → facebookexperimental/CUTracer. PR references in commit messages reflect the new org from #213 onward.
  • Wiki → docs/ — The GitHub Wiki content now lives in-repo under docs/ with an auto-sync workflow.
  • CONTRIBUTING.md — Reset to the standard GitHub contributing template.
  • Monthly stale-PR cleanup workflow — Adds a scheduled job to prune stale PR branches.
  • Per-test/per-run subdir layout — tests/py_add reorganized with TEST_KEEP_OUTPUT support for debugging.

CI

  • H100 CI lane (#219, #217) — A new self-hosted H100 workflow running the upstream Triton test suite, alongside the existing T4 test.yml (later retired below).
  • Triton-Nightly matrix leg (#218) — Replaces install-triton.sh with a matrixed nightly install; nightly pinned to the latest upstream commit (workaround for in-flight breakage).
  • Inductor inline-compile (#223) — test_add.py forces inductor inline compile to dodge an upstream Triton subprocess-pool driver crash.
  • Retired legacy cu128 T4 workflow (#226) — test.yml removed in favor of the new H100 lane.
  • OSS unit-test fix (#222) — Separates the Meta-internal unit tests so the OSS unit-test suite builds cleanly.
  • PyPI publish hardening — pypa/gh-action-pypi-publish pinned to a commit SHA in nightly-pypi.yml.

🐛 Notable Bug Fixes

  • NDJSON heap double-free (#225) — Concurrent std::string::append on the shared json_buffer_ triggered libstdc++ _M_mutate double-free under ASan; surfaced in OSS CI as random "unexpected character" JSON decode errors at ~7% per kernel-run on aarch64. Fixed by serializing write_trace with std::mutex.
  • uregs_indices schema bound — Raised from 62 to 255 so new Blackwell architectures don't break validate.

⚠️ Breaking Changes

  • CUTRACER_JSON_ENGINE env var removed — The opt-in switch from the migration window is gone now that RapidJSON is the default and the nlohmann per-record path has been deleted. Output is byte-identical for reg/mem/opcode/mem_value records and semantically identical for tma_trace. No user action required unless you were setting CUTRACER_JSON_ENGINE=ab for serializer parity testing.
  • GitHub organization moved — Repository now lives at facebookexperimental/CUTracer. Update any pinned remotes / submodule URLs / CI references.
  • Legacy T4 CI workflow retired — Anything that referenced the test.yml workflow name in T4-targeted automation should switch to the new H100 workflow.

Everything else is additive (new flags, new schemas).

🙏 Acknowledgments

CUTracer is built on NVBit by NVIDIA Research. We thank the NVBit team for their excellent binary instrumentation framework — and in particular for the NVBit 1.8 TMA APIs that let this release retire CUTracer's non-public-ISA TMA workarounds.

Contributors to this release: Chen Li, Lei Wang, Xu Zhao, Yueming Hao (alphabetical).

📄 License

  • MIT License — Meta Platforms, Inc. contributions
  • BSD-3-Clause License — NVIDIA NVBit components

See LICENSE and LICENSE-BSD for details.

📚 Documentation

Full documentation is now in-repo under docs/ (auto-synced to the Wiki).

🔗 Links

Assets 2
Loading