Lightning Talk: Tensor and 2D Parallelism
summary
Enables DDP + tensor parallel and a prototype pipeline + tensor parallel with activation-checkpointing that all-gathers shared inputs during recompute; adds sequence-dimension sharding for Transformer attention (column then row linear with final reduce-scatter). Cuts CPU overhead with overlapping collectives and torch.compile integration. Replaces TCP store with a nonblocking backend and scales startup/barrier to ~30,000–32,000 ranks (previous store fails around ~7,000), and adds process-group and collective hooks so operators can observe collectives, detect stragglers, hangs, and speed troubleshooting.
AI-written summary · about
chapters (5)
- 0:00Pyro distributed updates
- 1:29Sequence parallelism in Pyro
- 3:15Scaling large-rank training
- 6:25Debugging training jobs
- 8:32Q&A CPU vs GPU
related talks
08mPyTorch Distributed: Towards Large Scale Training
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
10mPyTorch Distributed Data Parallel (DDP)
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
10mLightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond
16mRevisiting Pipeline Parallelism for LLM Serving
34mPanel Discussion: Training and Inference at Planet Scale
26mTorchStore: What We Learned Building Distributed Storage Sol...