Architecting Multi-tenant Data-center Networks for GPU Customers
summary
Google’s A4 (B200) pods virtualize the GPU backend by programming CX7 NICs’ ASAP2 programmable block via the Docaflow SDK to run a PSP-based overlay. Measured overhead is tiny — about 2% throughput cost and negligible tail latency — while serving hundreds of terabytes and hundreds of millions of PSP encapsulation connections. A standby flow database enables hitless updates so upgrades avoid the ~20s per-NIC outage that would otherwise abort large RDMA jobs. The solution operates at scale and integrates telemetry and VPC trace tooling.
AI-written summary · about
chapters (8)
- 0:00The case for GPU virtualization
- 2:11Google A4 cluster overview
- 3:36Virtualization stack design
- 5:14Hardware-accelerated programming
- 7:56Performance and overhead metrics
- 9:52Troubleshooting with VPC traceroute
- 13:23Hitless software updates
- 16:08Summary and conclusion
related talks
05mDay 1 Keynote
23mG-NET: Effective GPU Sharing in NFV Systems
14mWhy We Don’t Need More Data Centers
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
21mExperience Operating Large GPU Clusters at Organizational Scale
18mResiliency at Scale: Managing Google’s TPUv4 Machine Learning Supercomputer
29mThe Path to GPU as a Service in Kubernetes
07mWelcome Remarks