Fsdp advanced tutorial (pytorch#1959)

* Adding FSDP advanced tutorial on T5 training

* FSDP advanced featues in progress

* removing old flow charts

* clean up

* clean up

* clean up

* clean up

* intro updates

* Updates on features

* typos correction

* Update sharding strategy

* adding cross ref

* Update FSDP_adavnced_tutorial.rst

* Update FSDP_adavnced_tutorial.rst

* added summary to the end

* added reference to code section for adding each feature

* update text grammar and minor details for top section

* removed activation checkpointing

* various grammar cleanup and minor explanatory additions (upper section)

* update transformer wrapping policy and mixed precision

* complete edits/updates for feature sections

* add bolded summary

* updated the backward prefetch

* update the example link

* Update FSDP_adavnced_tutorial.rst

* update authors

* minor spelling fixes

* typos fix

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* clean up

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* addressing comments

* addressed backward prefetch comments

* addressed the comment on model checkpoint saving

* Addressing the zero2 comments and overview of FSDP

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* updates based on comments

* FSDP wrapper comments

* updated with additional comments

* updated the bfloat16 memory

* updating the title separator

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* Update intermediate_source/FSDP_adavnced_tutorial.rst

Co-authored-by: Svetlana Karslioglu <[email protected]>

* addressing comments on code consistency

* remove CPUoffload import

* mixed precision re-wording

* addressing comments

Co-authored-by: Less Wright <[email protected]>
Co-authored-by: Svetlana Karslioglu <[email protected]>

Author: 3 people