TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed...
summary
TopoOpt co-optimizes computation, communication, and physical network topology to speed distributed DNN training, claiming up to 3.4x faster 99th-percentile iteration time versus cost-matched Fat-Tree. It uses alternating optimization between parallelization strategy and topology, exploits the mutability of AllReduce traffic to overlap permutation-based subtopologies, restricts candidates to regular permutation rings, and searches with a group-theory routine called TotientPerms. A 12-node RDMA-forwarding prototype at 100 Gbps and large-scale simulations validate the approach.
AI-written summary · about
chapters (8)
- 0:00The distributed training challenge
- 1:22Identifying network bottlenecks
- 4:29TopoOpt architecture
- 6:09DNA traffic characteristics
- 9:19Topology generation algorithm
- 11:06Hardware implementation
- 12:12Evaluation and results
- 14:29Conclusion and demo
related talks
16mOptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the...
17mTraffic Engineering for AI Training Networks
22mPerformance Optimizations at 100K+ Scale
14mEfficient Direct-Connect Topologies for Collective Communications
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
13mContracting Wide-area Network Topologies to Solve Flow Problems Quickly
15mForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics
16mAutomatic Parallelization of Software Network Functions