OptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the...
summary
OptiReduce is a time‑bounded AllReduce system that trades guaranteed delivery for bounded loss and faster completion: it implements an unreliable bounded transport, per‑round adaptive timeouts, a peer‑to‑peer Transpose AllReduce (TAR) to avoid inline aggregation loss, and randomized encoding via Hadamard transform. Built and evaluated on CloudLab and local clusters (A30/V100 GPUs, up to 24 real nodes and 144‑node simulations), it reaches target accuracy faster — e.g., 70% and 30% faster TTA vs Gloo and NCCL on average, TAR reduces 8‑node MSE by 7× vs ring, and it preserves convergence while delivering 2–2.3× speedups under high tail conditions.
AI-written summary · about
chapters (10)
- 0:00OptiReduce and cloud training
- 1:42Distributed data parallel training
- 3:07Tail latency in AllReduce
- 4:11Naive unreliable approaches
- 7:02OptiReduce design for tail latency
- 8:48Boundary transport and timeouts
- 9:53Transpose AllReduce algorithm
- 11:16Randomized transforms for accuracy
- 12:19Evaluations and performance
- 14:47Conclusion and release
related talks
17mTopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed...
16mSpotnik: Designing Distributed Machine Learning for Transient Cloud Resources
12mElastic Resource Sharing for Distributed Deep Learning
29mDistributed Deep Learning with Horovod
19mSwing: Short-cutting Rings for Higher Bandwidth Allreduce
12mATP: In-network Aggregation for Multi-tenant Learning
19mOptimizations and abstractions for sparse machine learning
08mMaximizing ROI Through Distributed Computing and Multi-Cloud