PyTorch Distributed: Towards Large Scale Training

on YouTube ↗08m

" summary

PyTorch Distributed describes concrete scaling features: data parallelism via DDP (replicate+allreduce) and FSDP (parameter sharding with all-gather), FSDP adds safe-tensor hooks for activation checkpointing, BFloat16 mixed precision, and a forward prefetch rate limiter to reduce OOMs. Dynamo works with DDP through torch.compile by breaking graphs at collective boundaries to preserve overlap. Pipeline parallelism uses Pippy with deferred init and automatic split heuristics; tensor parallelism uses parallelized_module/DTensor for column/row and 2D sharding.

AI-written summary · about

" chapters (6)

1. 0:00Introduction and mission
2. 1:18Distributed API architecture
3. 2:01Data parallelism and Dynamo
4. 3:57Pipeline parallelism with Pippi
5. 5:17Tensor and 2D parallelism
6. 7:06Conclusion and resources

" related talks

• 24mTraining large scale models using PyTorchShagun Sodhani · PyData Global 2023
• 19mComposable Distributed PT2(D)Wanchao Liang · PyTorch Conference 2023
• 10mPyTorch Distributed Data Parallel (DDP)Pritam Damania · PyTorch Developer Day 2020
• 14mLightning Talk: Large-Scale Distributed Training with Dynamo and...Yeounoh Chung, Jiewen Tan · PyTorch Conference 2023
• 38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloadsMansi Agarwal · DevConf.IN 2026
• 27mTips and tricks for distributed large model trainingVaibhav Singh, Nikita Namjoshi · Google I/O 2022
• 24mFast training and near-linear scaling with DataParallel in Amazon SageMakerAditya Bindal · AWS re:Invent 2020
• 11mLightning Talk: Tensor and 2D ParallelismRodrigo Kumpera, Junjie Wang · PyTorch Conference 2023