Resilient AI Supercomputer Networking using MRC and SRv6
summary
A new RDMA transport called MRC sprays packets across many paths and uses SRv6 source routing so hosts bypass failures without switch control, handling out‑of‑order packets by carrying GPU buffer addresses. Production runs at OpenAI on 4x200 and 8x100 fabrics (400/800Gbps) train frontier models; a 75,000‑GPU job survives a zombie T1 with ~25% of QPs affected, ~0.5M dropped packets, and full throughput recovery within a few RTTs. MRC ships from NVIDIA, AMD, Broadcom and uses 32 active QPs with backups.
AI-written summary · about
chapters (10)
- 0:00Resilient AI networking intro
- 1:31MRC transport protocol
- 5:44RoCE struggles with AI training
- 7:28Packet spraying for capacity
- 9:12Handling out-of-order packets
- 10:56SRv6 source routing
- 13:48SRv6 path selection
- 17:01MRC dynamic failure recovery
- 18:26Real-world failure recovery
- 20:54MRC status and future
related talks
17mMeta’s Network Journey to Enable AI
21mScaling RoCE Networks for AI Training
19mAI/ML Networking Challenges: The Fast and the Finicky!
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
15mLive Q&A with Speakers – Joseph Provine
19mScaling AI Network with DSF
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
18mResiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer