Maintaining Large Scale AI Capacity
summary
Meta describes moving from one research cluster to nearly a hundred AI superclusters and targeting 600,000 GPUs while running 1,000,000 operations/day. A central orchestrator Ops Planner serializes maintenance, rolling driver and firmware updates, and enforces gold/silver/bronze upgrade guarantees. Teams add deep health checks with NVIDIA, run three intrusive health probes per host, carve fleets into maintenance domains and trains to limit interruptions, and report a billion maintenance operations to date with rolling upgrades working across diverse GPUs, networks and drivers.
AI-written summary · about
chapters (11)
- 0:00Meta's AI infrastructure
- 1:54600k GPUs by year-end
- 3:411M daily operations maintenance
- 7:13Traditional AI maintenance limits
- 8:38Rolling upgrades strategy
- 10:49Ops Planner orchestration
- 12:34Hardware uniformity challenges
- 13:56AI hardware diversity
- 15:00Minimizing AI training interruptions
- 16:03Generative AI infrastructure
- 17:26Future vision commitment
related talks
28mScaling agentic AI: From infrastructure to enterprise outcomes
12mKeynote: AI Infrastructure- 28mScaling agentic AI: From infrastructure to enterprise outcomes – Kevin Cochrane
19mScaling AI Network with DSF
23mKeynote
22mPerformance Optimizations at 100K+ Scale
10mApproach to AI at a large scale
18mThe AI-First Data Infrastructure