Di-PS: System-Algorithm Co-Design for Asynchronous and Heterogeneous Cross-cluster...
summary
Di-PS presents a centralized leader-follower PS architecture plus algorithmic controls that reduce cross-cluster stalls and stabilize asynchronous updates. The system uses a pseudo-gradient penalty (aka pseudo-gradient correction) to filter low-quality updates, overlaps communication/computation, and uses heartbeat-based elasticity for fault isolation. Evaluation shows 1.27–4.67× speedups over synchronous baselines, >98% scaling efficiency, production deployment across 9 clusters and >10,000 NPUs, and a 100B-parameter LLM trained with ~6% overhead over single-cluster runs.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:02LLM training trends
- 2:50Why cross-cluster training
- 3:34Current cross-cluster limits
- 4:59Cross-cluster challenges
- 7:45Parameter server advantages
- 8:28DPS design requirements
- 9:12DPS architecture
- 9:53Scalable communication
- 10:55Convergence with gradient correction
- 12:00Resilience via heartbeat elasticity
- 12:42Evaluation and results
- 14:49Summary
related talks
09mHyperParallel: A SuperPoD-Aware Distributed Acceleration Library
17mZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
14mPollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
25mAvoiding Overload in Distributed Systems
14mAccelerating PyTorch FSDP Via Overlapping Collectives With In...
19mCrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
17mDuhu: Shared Disaggregated Memory for Distributed Data Processing Frameworks