Skip to content

Ulysses-SP WO A2A SAC fix - #4553

Draft
AlbedoWang wants to merge 1 commit into
pytorch:mainfrom
AlbedoWang:kaijian/ulysses-sp-wo-a2a-sac-fix-upstream
Draft

Ulysses-SP WO A2A SAC fix#4553
AlbedoWang wants to merge 1 commit into
pytorch:mainfrom
AlbedoWang:kaijian/ulysses-sp-wo-a2a-sac-fix-upstream

Conversation

@AlbedoWang

Copy link
Copy Markdown

Summary

  • Save the AutoParallel Ulysses-SP attention output A2A and post-WO boundary under GraphTrainer eager SAC.
  • Prevent backward rematerialization of the A2A and WO while preserving the existing eager SAC linear counter.
  • Scope the rule to the realized A2A-to-linear-to-residual graph using module FQN metadata.

Why

The row-sharded WO-to-reduce-scatter path already saves its logical output through the collective policy. The Ulysses-SP A2A-to-WO path did not have an equivalent save boundary, so backward recomputed one A2A and one WO per transformer layer.

Validation

  • Focused GraphTrainer AutoParallel tests: 9 passed on the PR branch.
  • Full validation on the original experiment branch: 14 AutoParallel tests passed; 202 graph-pass tests passed and 1 skipped.
  • A 32-H100 paired run reduced target WO A2A calls from 64 to 32 and eliminated all 32 recomputed calls.
  • Patched 4K fresh measured 287.678 ± 0.334 ms/step versus 295.419 ± 0.863 ms/step for 2K replay on the same allocation.
  • Patched 4K fresh used 6.724 GiB active memory versus 6.667 GiB for 2K replay.

Known issue

Strict bitwise parity between patched and unpatched fresh runs did not pass. The step-2 relative differences were 2.94e-5 for loss and 1.29e-4 for grad norm, so the change remains draft while this numerical drift is investigated.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant