Composable Distributed PT2(D)
summary
PyTorch Distributed adds three composable pieces: Device Mesh for automatic process-group setup across hosts and GPUs, DTensor for tensor-level sharding/replication and uniform sharded state, and distributed checkpoint that saves sharded state without redundant copies and can load into a different world size. The repo includes a native 2D parallel API (tensor parallel + FSDP), an HSDP mode, torch.compile compatibility, and nightly builds claiming competitive end-to-end performance for large‑model pretraining.
AI-written summary · about
chapters (9)
- 0:00Composable PyTorch distributed training
- 1:08Scaling large language model training
- 2:33Device Mesh for multi-dimensional parallelism
- 5:04D Tensor as tensor sharding
- 7:11Two-dimensional parallel API
- 10:20Hybrid sharded data parallel (HSDP)
- 11:23Distributed checkpoint for state management
- 13:09Torch Compile compatibility
- 14:55Q&A
related talks
08mPyTorch Distributed: Towards Large Scale Training
10mPyTorch Distributed Data Parallel (DDP)
24mSponsored Session: PyTorch Distributed and Fault Tolerance
24mTraining large scale models using PyTorch
27mTips and tricks for distributed large model training
09mHyperParallel: A SuperPoD-Aware Distributed Acceleration Library
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
23mSponsored Session: Powering PyTorch: Decentralized Training for an Energy-Hungry Future