[docs] Fix leftover snapshot paths, metrics dual-log, and model guide - #4563
[docs] Fix leftover snapshot paths, metrics dual-log, and model guide#4563YeonwooSung wants to merge 2 commits into
Conversation
debugging.md still described pre-move memory snapshot paths. metrics.md said W&B wins over TensorBoard; both loggers attach. checkpoint.md had one convert_from_hf example on the old scripts/ path. models/README.md still mentioned apply_moe_ep_tp, a train.py CP context manager, and per-model pipeline.py.
meta-cla
Bot
added
the
CLA Signed
| - apply training techniques in the following order | ||
| - `model.parallelize(parallel_dims)` — auto-recursive declarative sharding driven by `sharding_config` (TP, SP, attention `local_map`). Replaces per-model `parallelize_module` plan dicts. | ||
| - (MoE models) `apply_moe_ep_tp` for expert-parallel + TP on MoE experts (not yet config-based). | ||
| - (MoE models) expert-parallel sharding via `apply_fsdp_to_decoder` plus `sharding_config` / expert meshes (there is no `apply_moe_ep_tp` helper). |
There was a problem hiding this comment.
remove description of things that doesn't exist, aka "(there is no apply_moe_ep_tp helper)"
There was a problem hiding this comment.
Rephrased in 37bae3d to describe what is there rather than what is not: model.parallelize applies the EP axes declared in sharding_config, and apply_fsdp_to_decoder takes ep_degree / edp_mesh for the expert FSDP mesh.
| - apply PP | ||
| - NOTE: language-model CP goes through `Decoder.preprocess_inputs` -> `prepare_context_parallel_input`. Ideally no extra work is needed to enable CP. | ||
| - Pipeline parallelism (optional if model size is small) | ||
| - apply PP via `pipeline_llm` / `pipeline_vlm` in `torchtitan/distributed/pipeline_parallel.py` (no per-model `pipeline.py`) |
There was a problem hiding this comment.
this will be updated soon, so let's not change
There was a problem hiding this comment.
Reverted the pipeline.py section to its current text in 37bae3d, leaving it for the upcoming update. The CP NOTE just above it is still changed -- Decoder.preprocess_inputs calls prepare_context_parallel_input and there is no longer a CP context manager in train.py. Happy to drop that line too if it is part of the same upcoming change.
Describe the MoE expert sharding path positively instead of noting the absence of apply_moe_ep_tp: model.parallelize handles the EP axes from sharding_config, and apply_fsdp_to_decoder takes ep_degree / edp_mesh. Revert the pipeline.py section, which is being updated upstream.
2 participants
Summary
Leftover docs that no longer match
mainafter the recent README / script-path cleanups:docs/debugging.mdstill described memory snapshots as./outputs/memory_snapshot/iteration_x. Code defaults areprofiling/memory_snapshot/step_{step:012d}(and_exiton OOM) under the dump folder.docs/metrics.mdsaid W&B wins if both backends are enabled._build_metric_loggerattaches both loggers.docs/checkpoint.mdhad oneconvert_from_hf.pyexample still on the oldscripts/path.torchtitan/models/README.mdstill mentionedapply_moe_ep_tp, a CP context manager intrain.py, and per-modelpipeline.py.Docs only. No runtime change.
Test plan
profiler.py,metrics.py, conversion scripts, and current model/PP/CP entry points