Swing: Short-cutting Rings for Higher Bandwidth Allreduce

on YouTube ↗19m

" summary

New algorithm Swing shortens allreduce paths by short-cutting rings and swinging between torus directions, splitting each message so every node uses all local ports and alternates send directions to cut flow collisions. Evaluation uses SST Bucket Level simulation with 400 Gbit/s links and ~100 ns link latency, scales to 16k nodes, and shows up to 3x faster (≈+120%) for 32B–128MiB vectors, while the largest regression is about −22% for very large reductions.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to Swing
2. 1:10Torus networks in deep learning
3. 2:45State of the art and challenges
4. 7:16Algorithm intuition
5. 10:10Efficient port utilization
6. 12:26Evaluation and conclusion

" related talks

• 16mOptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the...Lalith Suresh, Muhammad Shahbaz, Matty Kadosh, Yonatan Piasetzky, Shay Vargaftik, Ertza Warraich, Omer Shabtai, Khalid Manaa · NSDI '25
• 45mFaster or better designed? Choose any two!Dmytro Vyazelenko · Voxxed Days Zürich 2024
• 38mLost in TransductionChristophe Grand · :clojureD 2017
• 53mMapReduce AlgorithmicsSergei Vassilvitskii · Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013)
• 16mMinFlow: High-performance and Cost-efficient Data Passing for I/O-intensive Stateful...Tao Li, Yongkun Li, Yinlong Xu, John C. S. Lui, Wenzhe Zhu · 22nd USENIX Conference on File and Storage Technologies (FAST '24)
• 54mReduction theorems for effective parameterized verification of round- based distributed algorithmsNathalie Bertrand, Pranav Ghorpade, Sasha Rubin · FRIDA 2025
• 16mSöze: One Network Telemetry Is All You Need for Per-flow Weighted Bandwidth Allocation...Weitao Wang, T. S. Eugene Ng · OSDI '25
• 25mApproximating Fair Queueing on Reconfigurable SwitchesMing Liu, Arvind Krishnamurthy, Naveen Kr. Sharma, Kishore Atreya · NSDI '18