Traffic Engineering for AI Training Networks
summary
Meta runs RoCE training clusters with up to 4K GPUs in a two-layer “factory” topology (RTW/CTSW) where each RTW has 16 downlinks; static DPR routing caused big performance variance when jobs are fragmented or links fail. A centralized TE controller ingests job placements and Open/R topology, runs a constraint shortest-path-first load‑balancer, and programs Broadcom exact-match programming into switches with NIC-to-host granularity, falling back to BGP and achieving roughly ≈50 ms convergence. Simulation and Nickel hardware benchmarks show TE gives consistent, near‑optimal completion times versus large regressions under DPR.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:59ROCE based Training Cluster Topology
- 2:14Al Workload Challenges
- 3:26Deterministic Path Routing (DPR)
- 4:17Performance Inconsistency over Job Placement
- 6:01Performance Inconsistency over Failures
- 7:46Control Plane
- 8:58Data Plane
- 9:51Challenge: Flow Granularity - Nic To Host Flow
- 10:59TE - Handle Network Failures System designed to operate headlessly
- 12:48Platform 1 - Simulation
- 13:38Platform 2 -NCCL Benchmark
- 14:13Job Placement Performance - Case Study
- 15:14TE's Challenges
related talks
21mDC Networks for Generative AI
17mDOTE: Rethinking (Predictive) WAN Traffic Engineering
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
17mPrecise Data Center Traffic Engineering with Constrained Hardware Resources
19mAI/ML Networking Challenges: The Fast and the Finicky!
22mMetaRoCE: Meta’s RDMA Transport
20mMetaRoCE: Meta’s RDMA Transport
17mTopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed...