ZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
summary
Claims a near‑optimal sparse-gradient synchronization that cuts communication up to 5.09x and boosts training throughput up to 2.48x vs prior work, with microresults like 6.8x comm speedup on LSTM and ~3.5x when using DGC top‑5% sparsification. Implements a previously unrealized balanced parallelism design (point‑to‑point, one‑shot aggregation, tensor partitioning) and realizes it on GPUs with a hierarchical hashing scheme (multi‑hashing, per‑thread parallel/serial buffers, cross‑GPU consistent hashing) to guarantee balanced traffic and lossless aggregation.
AI-written summary · about
chapters (10)
- 0:00Zen introduction
- 1:25Distributed training basics
- 3:52Sparse tensor challenges
- 4:13Zen’s sync system
- 5:21Sync scheme design space
- 7:29Balanced vs centralized schemes
- 8:33Balanced parallelism on GPU
- 10:37Hashing for node balancing
- 14:58Performance results
- 16:25Summary
related talks
01h 14mDistributed and concurrent optimization for machine learning
26mDeepSpeed: Efficient Training Scalability for Deep Learning
19mOptimizations and abstractions for sparse machine learning
17mAccelerating Collective Communication in Data Parallel Training across Deep Learning...
01h 05mTutorial: Federated Optimization, Part III
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
18mTC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs
23mSparse Computing Drives Energy-Efficient Computation for Artificial Intelligence