AI/ML Networking Challenges: The Fast and the Finicky!
summary
Shows that DeepSpeed/PyTorch LLM training hinges on RDMA fabrics like RoCE or InfiniBand, with collective comms mapped to per‑Q‑pair sockets; demo cluster hits ~1.47 Tbps aggregate and ~91–92 Gbps per NIC, in‑rack one‑way delays ~500–600 ns rising to 3.1–3.4 µs under load. A single nick flapping kills the job (links can flap >10×/hr), forcing fallbacks to ~30‑minute checkpoints; a probe‑mesh clock‑sync plus load‑balancing fix restores throughput and auto‑recovers in the demo. Q pair visibility is critical.
AI-written summary · about
chapters (11)
- 0:00Introduction to AI networking
- 1:27AI workload layers explained
- 3:15Traditional vs AI networks
- 6:26Lack of visibility in AI networks
- 9:17Demo of one-way delay measurement
- 11:04Link flapping reliability issues
- 12:31Demo of NIC flapping impact
- 14:21Auto recovery solution demo
- 15:24Predictable performance challenges
- 16:49Load balancing solution demo
- 17:10Key takeaways and predictions
related talks
25mResilience for AI Workloads at Scale: The Fast and the Finicky!
30mThe Gashlycrumb Tinies of AI Networking You Must Know (or Languish!)
20mTurbocharging AI/ML workloads: Revving Up Speed and Resilience
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
25mHigh Performance Networking for Distributed DL Training in Production K8s
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
18mCo-Designing Communication for AI Accelerators
18mNetwork Observability for AI/HPC Training Workflows