Resiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
Describes a production 4096-node accelerator cluster built as 4096-node TPUv4 supercomputers with custom 3D torus-like fabric and per-chip ICI switches, managed by SDN and optical circuit switching that dynamically reconfigures port-to-port circuits. Software detects chip, link and board faults, triggers wild-first-hop + dimension-ordered routing and automated repair workflows, and reports empirically that the system reaches 99.98% availability, with hardware outages affecting about 1% of training jobs.
AI-written summary · about
chapters (7)
- 0:00TPU supercomputers at Google
- 3:54Reliability challenges at scale
- 7:46Circuit switches for resource allocation
- 10:16ICI networking protocol
- 12:23Job scheduling workflow
- 15:53Fault tolerance and routing
- 17:38Summary and Q&A
related talks
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
18mArchitecting Multi-tenant Data-center Networks for GPU Customers
17mMeta’s Network Journey to Enable AI
22mResilient AI Supercomputer Networking using MRC and SRv6
31mLive Panel: Scale-Out Networking for AI Supercomputers
15mAlibaba HPN: A Data Center Network for Large Language Model Training
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...