Towards Domain-Specific Network Transport for Distributed DNN Training
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
MLT is a domain-specific transport for distributed DNN training that implements a 3-step progressive scheme: (1) per-packet load balancing across multiple paths to eliminate hotspots without worrying about reordering, (2) priority queueing and selective dropping by layer and gradient magnitude to preserve convergence, and (3) bounded-loss tolerance that lets some gradient packets be lost without harming final accuracy. Prototype runs on commodity switches, plugs into TensorFlow/MXNet/PyTorch and PS/Ring All-reduce, and shows up to 62.2% faster end-to-end training in testbed and simulation.
AI-written summary · about
chapters (10)
- 0:00Domain-specific transport for AI
- 2:28Data-parallel DNN challenges
- 3:53AI-centric networking needs
- 5:01DNN workload observations
- 7:53MLT design principles
- 10:46MLT system overview
- 13:41Prototype implementation
- 15:11Testbed setup
- 16:24Performance results
- 17:52Large-scale simulation
related talks
11mScaling Distributed Machine Learning with In-Network Aggregation
19mAI/ML Networking Challenges: The Fast and the Finicky!
08mBuilding Production-Ready ML-Based Network Routing
12mATP: In-network Aggregation for Multi-tenant Learning
34mSpeeding up Your ML Workload: Pytorch Compile and Distributed Training
21mOperationalizing ML Training Infra at Meta Scale
30mApache MXNet 2.0: Towards Synergistic ML and DL with Standardization
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d