Networking Optimizations for Multi-Node Deep Learning on Kubernetes
summary
Shows concrete steps and measurements to get multi‑node DL running at near line rate on Kubernetes using SR-IOV, RoCE, GPUDirect and Multus. Reports that without PFC/ECN, missing kernel modules, wrong PCI/GPU affinity and no source‑based routing a 48‑GPU (five‑box) ResNet‑50 run loses roughly 2× performance; after fixes each NIC hits ~90% of 100Gbps. Uses Horovod style collectives, SR‑IOV device plugins, and source‑based routing for per‑NIC rings.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:04Machine Learning
- 1:19Deep Learning
- 1:46The Very Basics of Neural Networks
- 3:50Training and Inference
- 4:58Training Challenges
- 6:30Training Scale Out - Data Parallelism
- 7:48Networking For Scale Out Training
- 10:09GPUDirect RDMA Technology
- 15:02Networking Optimizations
- 16:06Priority Flow Control, Explicit Congestion Notification etc
- 17:54PFC/ECN
- 18:44Super Connect - switching architecture
- 22:01Source Based Routing to enable 3
- 23:18Performance/scale measurements
- 26:09Summary and Conclusion!
related talks
25mHigh Performance Networking for Distributed DL Training in Production K8s
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
27mMake Kubernetes Networking Ready for World Class AI and HPC ...
34mDemocratizing Machine Learning on Kubernetes [I]
42mPractice of Building AI Training Cluster Based on Kubernetes+RoCEv2
21mScaling RoCE Networks for AI Training