PyTorch Distributed Data Parallel (DDP)
summary
Overview of concrete DDP and c10d changes: DDP communication hook lets Python callables replace the synchronized all-reduce (useful for fp16 compression or gossip SGD), gradient_as_bucket_view eliminates the extra gradient copy so large models avoid a full extra GB of memory, and model.join supplies dummy syncs to handle uneven batch counts. Dynamic bucketing reorders based on first backward and yields 3–7% speedup on BERT/RoBERTa. Also adds RPC+DDP composition, Windows c10d, and upcoming GPU RPC, point-to-point and pipeline parallelism.
AI-written summary · about
chapters (13)
- 0:00Start
- 0:19Agenda
- 0:36Refresher of Distributed Data Parallel
- 1:26Ddp Communication Hook
- 2:14Support for Uneven Inputs in Ddp
- 3:19Memory Optimizations for Ddp
- 4:03Combining Ddp and Rpcs
- 5:12Dynamic Bucketing in Ddp
- 6:42What's Coming Soon in Pytorch
- 7:28Pipeline Parallelism
- 8:58Auto Tuning for Ddp
- 9:15Hybrid Parallelism
- 9:54Distributed Overview on Python
related talks
08mPyTorch Distributed: Towards Large Scale Training
19mComposable Distributed PT2(D)
24mFast training and near-linear scaling with DataParallel in Amazon SageMaker
11mLightning Talk: Tensor and 2D Parallelism
24mTraining large scale models using PyTorch
09mHyperParallel: A SuperPoD-Aware Distributed Acceleration Library
14mAccelerating PyTorch FSDP Via Overlapping Collectives With In...
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...