The GPUs on the Bus Go ‘Round and ‘Round
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 16:15· Level 1 | Hall Entrance S10 | Room B·
summary
NVIDIA GPU Cloud runs 60,000+ GPUs across 40+ clusters and sees roughly 120–200 GPU failures per 24 hours (about 0.6% of devices), so reliability is operationally critical. Detection uses a custom GPU problem detector (device plugin) that sets node conditions consumed by node-problem-detector, triggering an automated remediation pipeline: reboot→power-cycle→rebuild with an alert for remediation loops (>2 reboots in 90 minutes). Reboots fix many cases but node drains can take up to eight hours; PA and other automation reduce manual effort and improve capacity.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:08NVIDIA’s Kubernetes clusters
- 3:13GPU device failures
- 6:24GPU failures impact
- 7:52Automated failure remediation
- 10:06Reboot as remediation
- 12:18Manual to automated processes
- 13:27Monitoring GPU issues
- 16:44Recovery workflow
- 19:15Proactive maintenance
- 20:40GPU density challenges
- 22:01Future challenges
- 23:27Node Lifecycle Working Group
- 24:07Q&A
related talks
21mExperience Operating Large GPU Clusters at Organizational Scale
29mThe Path to GPU as a Service in Kubernetes
15mEnsuring 100% uptime in the cloud of failing GPUs
21mSponsored Session: Fault-Tolerant Training: How We Build Rel...
14mWhy We Don’t Need More Data Centers
24mClosing Keynote
18mArchitecting Multi-tenant Data-center Networks for GPU Customers
31mAI, CERN, and the Quest for GPU Custody: How CERN Leverages DRA for Efficient GPU Sharing