High Performance Networking for Distributed DL Training in Production K8s
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Wednesday, 2020-11-18· at 15:00· Intrado Virtual Event Platform,·
summary
An 800‑GPU production cluster (100 DGX‑1 nodes) runs RoCEv2 rings for GPU‑to‑GPU traffic and isolates storage/control on dual 10G Ethernet; pods get RDMA VFs via the SR‑IOV device plugin and Multus delegates to SR‑IOV CNI + Flannel so NCCL can build one‑hop rings that avoid output‑port congestion. A nickel all‑reduce preflight shows ~45 Gbps average per 50 Gbps link and PyTorch training scales near ideal up to 256 GPUs under the custom batch scheduler and MPI operator.
AI-written summary · about
chapters (8)
- 0:00Start
- 6:09Perimeter Network
- 12:05Design Considerations for Multi-Node
- 14:15Control Plane
- 15:12Custom Job Controller
- 17:04Topology of a Dgx1
- 21:16Compute Phase
- 23:15Conclusion
related talks
34mNetworking Optimizations for Multi-Node Deep Learning on Kubernetes
34mNetworking Optimizations for Multi-Node Deep Learning on Kubernetes
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
42mPractice of Building AI Training Cluster Based on Kubernetes+RoCEv2
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
34mSupercharge Kubeflow Performance on GPU Clusters
19mAI/ML Networking Challenges: The Fast and the Finicky!
31mHigh-Performance MPI and Deep Learning on OpenPOWER Platform