Auto-file GitHub issues for nightly UT new failures - #5119
Open
RUIJIEZHONG66166 wants to merge 4 commits into
Open
Auto-file GitHub issues for nightly UT new failures#5119RUIJIEZHONG66166 wants to merge 4 commits into
RUIJIEZHONG66166 wants to merge 4 commits into
Conversation
RUIJIEZHONG66166
force-pushed
the
agent/nightly-ut-auto-issue
branch
from
August 27, 2026 02:54
82fea5d to
5046519
Compare
RUIJIEZHONG66166
force-pushed
the
agent/nightly-ut-auto-issue
branch
from
August 27, 2026 04:00
5046519 to
2f0ec2e
Compare
New failures in the nightly UT run are today only visible as a red job and a CSV artifact, and muting one means a human hand-writing a skip issue. This adds a deterministic bot that files one issue per root cause, plus the artifacts it needs from the existing UT pipeline. Filing an issue mutes a test: fetch_issues.sh subtracts the `Cases:` block of every open `skipped` issue from the next night's failures. Every decision here therefore fails in the direction of leaving a case running rather than silently muting it, and the job stays in --dry-run until the repo variable UT_AUTO_ISSUE_ENABLED is set to `true`. Review in this order: 1. The producers. These only add artifacts and change no existing behaviour: check-ut.py writes all_cases_<category>.log (the full case universe, which unlike passed+failures also carries skipped cases), ut_result_check.sh records its per-category count verdict to run_health.jsonl, and the two composite actions land collect_env.log and versions.txt under ut_log so the shas actually under test travel with the artifact. 2. The wiring: _linux_ut.yml exports UT_RUN_HEALTH_FILE, _ut_auto_issue.yml is a new callable workflow, and nightly_ondemand.yml calls it after the UT matrix - on !cancelled() rather than success(), since the UT job fails exactly when there is something to file. 3. ut_auto_issue.py. Its stage banners, Stage 0 through Stage 8, are the intended reading order. 4. ut-auto-issue-body.md, whose `Cases:` block and trailing marker are both machine-consumed; its header comment says which consumer reads what. Three choices had a real alternative and are worth calling out. Grouping keys on (normalized error, test file) rather than on the error alone. The error alone would collapse a generic message - a precision mismatch, or the sentinel used when a case dies without one - across unrelated files into a single issue and therefore a single very wide mute. Over-splitting is the safe direction, so one root cause spread over several files stays several issues, and they are linked to each other by a comment instead of being merged. The baseline is resolved per category, not per run. There is no "last good nightly", only "the last nightly in which this category completed healthily": a run that truncated in op_ut is still a perfectly good op_extended baseline. The per-run issue cap admits or defers a whole error family at once. A plain head/tail cut on the size-ordered list would file the big test files of one root cause and defer its small ones, muting half a bug and leaving the rest to fail every night. Test Plan No automated test ships with this change, so the pre-enable dry run below is the gate, not a formality. During development the pure logic was exercised against stubbed inputs - normalization stability across nights, grouping, the family-atomic issue budget, the Cases: parse/append contract that does the actual muting, and a main() smoke test with every gh call replaced by a stub. Those harnesses were ad hoc and are not part of the change. Before UT_AUTO_ISSUE_ENABLED is set, the job must be dry-run against real past nightlies and the rendered bodies in the report artifact reviewed by hand; the normalization rules are the part most likely to need tuning: ```bash python3 .github/scripts/ut_auto_issue.py --run-id <past_nightly_run_id> --dry-run ``` That command was not executed here: this workstation has no authenticated gh and the script needs a real run id. The workflow's own default is --dry-run, so merging this changes no issue state until the repo variable is set. Authored with the assistance of an AI coding agent.
A nightly dry run against run 32733042537 classified `third_party.torch-xpu-ops.test.xpu.test_dataloader_xpu` as a new case failure. The file has been in the repo since #2989. Nothing about it is new. The row is not a test case at all. pytest reports a module that failed to import as a testcase with an empty classname and the dotted module path as the name, because _pytest/junitxml.py:mangle_test_address has nothing before the first `::` to put in classname. classify_case compares a case against its baseline by exact set membership, and a healthy baseline records that module's individual methods and never the module itself, so the row is in neither the passed nor the failed set and falls through to CLS_NEW_CASE unconditionally - for any file, of any age. Three things follow from that, in increasing severity. The label sends triage the wrong way: "upstream added a test we do not support yet" rather than "upstream deleted a symbol we import". Filing the issue mutes that one row, so the file stays dark every night with nobody told. And the cases the row hides do not fail, they vanish - a module that fails to import contributes nothing to passed_<cat>.log or failures_<cat>.log, so those cases never reach new_ut_failure_list.csv, and a few hundred missing cases sit far below the 5% count gate in ut_result_check.sh:check_test_cases. Review in this order. 1. Case.is_collection_error / Case.module / Case.test_file. The identity fix. `module` is the granularity the two other stages need, and deriving test_file from it also repairs a second bug: a collection error used to yield test_file "unknown", which is part of the grouping key and of `sig`, so unrelated files collapsed into one group and one mute. For a real case test_file is unchanged, so no existing signature moves and MARKER_VERSION stays put. 2. build_groups. Collection errors join the infra denylist in being quarantined: reported, never filed, never muted. Group.quarantined becomes Group.quarantine, a reason, because the report now has two things to say. 3. Stage 4b. collection_error_context answers "how much stopped running" from the baseline's per-module index, and record_vanished_modules asks the same question independently of any error row. 4. The main() wiring and the finish() rendering. classify_case itself is unchanged; it only gained a comment saying why. Three choices had a real alternative. Report-only rather than filing these as regressions. Looking the module up in the baseline does establish that its cases used to pass, so a `regression` label would be defensible and would get the issue triaged. But filing is what mutes, and muting a row that stands for an entire dark file is the worst outcome available here. The nightly stays red instead. The baseline artifact, not git history, answers "did this file exist before". File age is not case age, the rows are dotted module paths rather than repo paths, and the op_ut leg also runs upstream files that are not in this repo. The baseline is exact and is already downloaded. Module granularity is confined to collection errors. Widening classify_case to compare every case against its module would be wrong in the other direction: a dtype parametrization added upstream is a genuine new case even though its module is years old. One cost worth naming: Stage 4b now runs before the nothing-to-file exit, so a green nightly does a baseline walk it used to skip. That is deliberate. A green night is exactly the night a silently emptied file has to be caught, and the skip list is a plausible way to empty one - xpu_test_utils.py:launch_test turns it into `pytest -k "not ..."`, and deselected cases are absent from the JUnit XML entirely rather than recorded as skipped. Test Plan The pure logic was exercised locally against stubbed artifacts: the three rows this run actually reported, plus the boundaries the fix must not get wrong - a collection error on a file the baseline never saw is still reported as a new test file, a newly parametrized case in an old module is still new_case_failure, regression/persistent/skipped-then-run classification for real cases is byte-identical to before, a row parsed out of a log by check-ut.py:parse_log_file is not mistaken for a collection error, and the infra denylist still wins. Those harnesses were ad hoc and are not part of this change. The gate is the dry run, which was not executed here: this workstation has no authenticated gh. ```bash python3 .github/scripts/ut_auto_issue.py --run-id 32733042537 --dry-run ``` Two things to check in its report. test_dataloader_xpu should have left the issue list for the "Modules that produced no cases in this run" table, carrying the count of cases it used to pass. And that table should be short - if a routine skip-list edit fills it every night it needs a significance threshold, which this change deliberately does not guess at. lintrunner was not run: it is not installed on this workstation and there is no .venv to activate. The file parses, has no line over 100 characters, no tabs, no trailing whitespace and no non-ASCII. Authored with the assistance of an AI coding agent.
Whether the bot files anything took three negations to read: the repo
variable was compared with `!= 'true'` to produce `dry_run`, the workflow
passed `--dry-run`, and every write site in the script was guarded by
`if not args.dry_run`. Reading the chain to answer "does a nightly file
issues" meant composing all three, and it is easy to mistake the
comparison in nightly_ondemand.yml for a place where the variable is set
rather than read.
Same behaviour, stated positively: UT_AUTO_ISSUE_ENABLED == 'true' feeds
a `create_issues` input, which passes --create-issues, which each write
site checks directly. Unset variable still means nothing is filed.
This also removes a footgun in the command line. --dry-run had to be
added to be safe, so a bare `ut_auto_issue.py --run-id N` filed issues
for real, and filing mutes tests. --create-issues has to be added to be
destructive, so a bare invocation now only reports.
The reusable workflow's input is renamed and inverted along with it
(default false), and the report key and rendered wording follow: "dry
run" becomes "report only".
Test Plan
lintrunner was not run: it is not installed on this workstation and
there is no .venv in the repo root or its parent to activate. The file
compiles, has no line over 120 characters and no non-ASCII.
Syntax of all three files:
```bash
python3 -m py_compile .github/scripts/ut_auto_issue.py
python3 -c "import yaml; [yaml.safe_load(open(f)) for f in \
['.github/workflows/_ut_auto_issue.yml','.github/workflows/nightly_ondemand.yml']]"
```
All seven write sites - ensure_labels, the two still-failing
comment_issue calls, edit_body, create_issue for a group, the cross-file
comment_issue, and create_issue for the umbrella - were enumerated and
their guard context read individually to confirm none became reachable
without --create-issues:
```bash
grep -n "create_issue(\|comment_issue(\|edit_body(\|ensure_labels(" \
.github/scripts/ut_auto_issue.py
```
The flag semantics and the renamed report key were exercised directly,
asserting that a bare parse yields create_issues=False and that
report.json, stdout and GITHUB_STEP_SUMMARY agree on "report only"
versus "live". The harness was ad hoc and is not part of this change.
The end-to-end gate is unchanged and still pending: a report-only run
against a real past nightly, which needs an authenticated gh this
workstation does not have.
```bash
python3 .github/scripts/ut_auto_issue.py --run-id <past_nightly_run_id>
```
Note that the two earlier commits on this branch quote `--dry-run` in
their own Test Plan sections. Those commands are now spelled without a
flag; the messages were left alone rather than rewritten.
Authored with the assistance of an AI coding agent.
A report-only run against nightly 33207055708 filed nothing and said only that op_ut was infra breakage. The leg had two new failures, one of which matched the infra denylist, and 1/2 is above the 30% threshold, so H7 discarded the leg and took the other failure with it. The gate had no minimum sample size. At a 30% threshold a single infra-looking failure clears it on its own for any n <= 3: 1/1, 1/2 and 1/3 are all above 0.3. The night a leg produces two new failures is exactly the night the ratio carries no information, and that is when the gate was most likely to fire. It also cost nothing to fire it wrongly, in the sense that the gate adds no protection at that size. A case whose own error matches the denylist is quarantined by build_groups regardless - reported, never filed, never muted. H7 exists only to distrust the cases *around* those, which is a claim about the machine, and two data points cannot support it. Here the only thing it achieved was to hide a genuine-looking forward-AD failure in test_ops_fwd_gradients_xpu. So the ratio now applies only over INFRA_MIN_CASES or more failures. Below that the infra-looking cases are still quarantined individually and a note says the gate was not applied. 10 is where a lone infra failure sits at 10%, comfortably under the threshold, while a runner that actually lost its GPU produces far more than ten. The alternative was to lower INFRA_SIGNATURE_RATIO instead, which is backwards: the problem is not that 30% is the wrong share, it is that a share of two is not a measurement. Second change, in the report rather than the logic. H7 is the only gate that drops failures without rendering them anywhere - collect_leg returns an empty list, so the cases never reach report["quarantined"] and no body is written for them - and it recorded only the string "infra signature ratio (H7)". Reading the report could not answer which failures were discarded or which of them matched the denylist. It now records every dropped case with its error and its infra verdict, and finish() lists them under the skipped leg. `infra` becomes a set because the report needs the individual cases rather than a count, and because `c in infra` over a list would be quadratic in a leg that can carry up to ABORT_THRESHOLD cases. Test Plan lintrunner was not run: it is not installed on this workstation and there is no .venv in the repo root or its parent. The file compiles, has no line over 120 characters and no non-ASCII. ```bash python3 -m py_compile .github/scripts/ut_auto_issue.py ``` The gate was exercised directly on the two rows from run 33207055708 and on a synthetic leg large enough for the ratio to mean something. With n=2 and one infra match both cases now survive and no leg is skipped; with n=20 and twelve infra matches the leg is still discarded, all twenty dropped cases are recorded, twelve are flagged infra, and the rendered summary lists each one. The firing table was checked across n=1,2,3,4,9,10,20 to confirm a lone infra failure no longer trips the gate at any size while 4/10 and 9/20 still do. That harness was ad hoc and is not part of this change. The end-to-end gate is the report-only run that surfaced this, repeated against the same nightly: ```bash python3 .github/scripts/ut_auto_issue.py --run-id 33207055708 \ --test-type build-weekly \ --work-dir ~/auto_file_github_issue/ut-auto-issue \ --report-dir ~/auto_file_github_issue/ut-auto-issue-report ``` op_ut should no longer appear under skipped legs. The collection error on test_prims_xpu should be quarantined as such and reported with the count of cases that module stopped contributing; the forward-AD failure should be classified against its baseline and rendered. Authored with the assistance of an AI coding agent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nightly UT new failures are today only visible as a red job and a CSV artifact, and muting one means a human hand-writing a skip issue. This adds a deterministic bot (no model calls) that files one GitHub issue per root cause, plus the artifacts it needs from the existing UT pipeline.
Filing an issue mutes a test.
fetch_issues.shsubtracts theCases:block of every openskippedissue from the next night's failures. Every decision in this change therefore fails in the direction of leaving a case running rather than silently muting it, and the job stays in--dry-rununtil the repo variableUT_AUTO_ISSUE_ENABLEDis set totrue. Merging this changes no issue state.Review order
check-ut.pywritesall_cases_<category>.log(the full case universe, which unlike passed+failures also carries skipped cases);ut_result_check.shrecords its per-category count verdict torun_health.jsonl; the two composite actions landcollect_env.logandversions.txtunderut_logso the shas actually under test travel with the artifact._linux_ut.ymlexportsUT_RUN_HEALTH_FILE,_ut_auto_issue.ymlis a new callable workflow, andnightly_ondemand.ymlcalls it after the UT matrix on!cancelled()rather thansuccess(), since the UT job fails exactly when there is something to file.ut_auto_issue.py— its stage banners, Stage 0 through Stage 8, are the intended reading order.ut-auto-issue-body.md— theCases:block and the trailing marker are both machine-consumed; the header comment says which consumer reads what.Safety rails
MAX_ISSUES_PER_RUN(15)Design choices with a real alternative
Grouping keys on
(normalized error, test file), not the error alone. The error alone would collapse a generic message — a precision mismatch, or the sentinel used when a case dies without one — across unrelated files into a single issue and therefore a single very wide mute. Over-splitting is the safe direction, so one root cause spread over several files stays several issues, and they are linked to each other by a comment instead of being merged.The baseline is resolved per category, not per run. There is no "last good nightly", only "the last nightly in which this category completed healthily": a run that truncated in
op_utis still a perfectly goodop_extendedbaseline.The per-run issue cap admits or defers a whole error family at once. A plain head/tail cut on the size-ordered list would file the big test files of one root cause and defer its small ones — muting half a bug and leaving the rest to fail every night.
Before enabling
UT_AUTO_ISSUE_ENABLEDmust stay unset until the job has been dry-run against a few real nightlies and the rendered bodies in the report artifact reviewed by hand. The normalization rules are the part most likely to need tuning. The labelsskipped,skipped_bmg,regressionandnew_case_failuremust exist in the repo first —ensure_labels()hard-fails otherwise.Test: none (CI-tooling change with no device-side behaviour). Validation is the script's own dry-run mode against a real past nightly, which is also the gate before
UT_AUTO_ISSUE_ENABLEDis set:That command was not executed on this workstation — it needs a real nightly run id. During development the pure logic (normalization stability across nights, grouping, the family-atomic issue budget, and the
Cases:parse/append contract that does the actual muting) was exercised against stubbed inputs, including amain()smoke test with everyghcall replaced by a stub.Authored with the assistance of an AI coding agent.