Publish CUDA 13.4 nightly wheels - #22653
Open
shoumikhin wants to merge 9 commits into
Open
Conversation
shoumikhin
added
the
release notes: build
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22653
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit c602baa with merge base d1d5f40 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
rascani
approved these changes
Sep 9, 2026
Gasoonjia
approved these changes
Sep 9, 2026
Gasoonjia
left a comment
Gasoonjia
left a comment
Contributor
There was a problem hiding this comment.
Can you add a 13.4 ci?
shoumikhin
force-pushed
the
et-cuda-add-cu134
branch
from
September 9, 2026 18:03
59bcd9c to
8a33833
Compare
shoumikhin
added
the
ciflow/binaries/all
shoumikhin
force-pushed
the
et-cuda-add-cu134
branch
from
September 10, 2026 02:20
e9b5152 to
9029da6
Compare
PyTorch now publishes CUDA 13.4 nightlies for every Python version ExecuTorch builds (3.10 through 3.14), and Torch-TensorRT already targets cu134 in its own build matrix. A consumer building a cu134 delegate needs a matching ExecuTorch wheel to depend on, and today there is none. Add cu134 to the list of CUDA trains the release publishes. This is a small policy change on top of the resilient filter: cu134 is published when the shared matrix generator offers it, which it does today, and is skipped automatically if it ever stops, exactly like the other trains. No build wiring changes are needed, because the CUDA wheel workflow builds whatever the generator produces and the filter keeps. Also add a 13.4 build-compatibility job to the CUDA test workflow, so the newly published train is actually built and smoke-tested in CI rather than published unverified. The same change drops 12.6 from that matrix, since PyTorch stopped publishing CUDA 12.6 and the job can no longer install it. The build script already takes the CUDA version as an argument, so no script change is needed. Example, with the generator offering cu130, cu132 and cu134: before: publishes cu130 and cu132 after: publishes cu130, cu132 and cu134 Test plan: updated the pinned published-set test to include cu134. Ran the filter against a matrix offering all three CUDA 13 trains and confirmed it exits 0 and publishes cu130, cu132 and cu134. Full suite: 21 passed, 2 subtests. The CUDA test workflow parses and its build job now runs for 13.0 and 13.4.
Recognize CUDA 13.4 in installer and wheel architecture selection. Check the publication policy against both build guards and preserve the distinction between absent and incomplete CUDA trains. Tested the CI-script suite (314 passed, 15 skipped), focused CUDA filter tests, pinned lint, and Bash syntax. Assisted by Devmate.
CUDA 13.4 needs packages that are not available in the existing release and torchao series. Select the published matching nightly set only for CUDA 13.4, retain it through example installation, and keep the wheel torchao bound aligned. Seven focused installer and metadata tests pass, and three assertions fail against the previous code. Core dependency resolution passed for Python 3.10 through 3.14 on both Linux architectures. Full Linux installation and GPU validation remain for CI. Assisted by Devmate.
The test imported a metadata decoder that no longer exists. Decode the current structured payload before rewriting it to the legacy two-key shape, including the no-constants control, so both paths reach the intended loader checks. Syntax and scoped lint pass. GPU execution against the current runtime remains for CI. Assisted by Devmate.
Use the matching driver setup and keep the reusable workflow and action checkout on the same revision. Retain CUDA 12.6 and 13.0 coverage, and reject installations whose torch CUDA version differs from the requested build. The CI-script suite passed 324 tests with 15 skipped. New regressions fail on the previous matrix, workflow reference, and CUDA-version probe. Scoped lint, workflow lint, and Bash syntax pass. Actual builds and GPU execution remain for CI. Assisted by Devmate.
Select the same torchao series for both installer modes while leaving source-built torch unpinned. Exercise the full installer through its package-install step and check that the installed torchao satisfies the generated dependency bound. The new check fails for both architectures before the fix; eight dependency tests now pass.
Pin both the reusable workflow and its action checkout to the same upstream-reachable revision, including the test dependency lint fix.
The pinned torchao source cannot satisfy the CUDA 13.4 wheel pin. Keep explicit source builds free of that competing wheel requirement and use package metadata that accepts the source version. Test Plan: The full installer regression failed for both source-build flags before the fix. The CI-script suite now passes 326 tests and 54 subtests, with 15 skipped.
The compatibility image's older Conda C++ runtime aborts at process shutdown after a correct CUDA matrix multiplication. Update the CUDA 13.4 job to the runtime versions that passed the same-wheel comparison, and check the GPU result against CPU output before normal process termination. Test Plan: 327 CI-script tests and 59 subtests passed, with 15 skips. New workflow cases fail without the runtime setup and verify that a failed update prevents the build. The isolated GPU comparison reproduced the baseline abort and exited cleanly after a normal Conda runtime update. Full compatibility CI follows publication.
shoumikhin
force-pushed
the
et-cuda-add-cu134
branch
from
September 10, 2026 05:06
877f0d8 to
c602baa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CUDA 13.4 consumers need a matching ExecuTorch wheel. The publication filter, build detection, dependencies, and compatibility environment must all support it.
Fix
Publish
cu134when the shared generator offers it. Distinguish a missing CUDA train from an offered train with incomplete Python coverage. Add CUDA 13.4 build detection and architecture mappings.Select matching CUDA 13.4 nightly dependencies and retain them through example installation. Keep package metadata consistent with source-pinned torch and explicit torchao source builds. Preserve existing dependency selection for other CUDA versions and the CPU torchao variant on ARM.
Run CUDA 13.4 compatibility with a newer driver and C++ runtime. The image's older Conda runtime aborts during thread-local cleanup after matrix multiplication. Update that runtime normally and verify the GPU result against CPU output, including normal process termination. Preserve CUDA 12.6 and 13.0 coverage.
Pin the reusable workflow and its checked-out actions to the same prerequisite revision from pytorch/test-infra#8771. That prerequisite still needs approval.
Update the stale missing-weights regression to decode current metadata and exercise the legacy payload with and without constants.
Testing
The CI-script suite passed 327 tests with 15 skipped. Regressions exercise dependency selection, incorrect GPU results, and runtime-update failure. Scoped Python, shell, and workflow checks passed.
The final revision passed CUDA 12.6, 13.0, and 13.4 compatibility. CUDA 13.4 completed installation, dependency checks, a GPU result comparison against CPU, and normal process termination. Both CUDA unit suites passed, including the missing-weights regressions.
All ten final CUDA 13.4 wheel builds and uploads passed for Python 3.10 through 3.14 on x86_64 and ARM64. The x86_64 Python 3.12 smoke test ran delegated models with and without external weights and matched eager output. ARM64 smoke tests check packaging without GPU execution. All remaining CI checks finished without failures.
The prerequisite's CI passed. Its approval is the only remaining merge-readiness gate.
Assisted by Devmate.