Experience Operating Large GPU Clusters at Organizational Scale
summary
Nvidia describes operating multi-tenant GPU clusters at ~95% occupancy by combining multi-level fair sharing (account, quota, user, workload) with minimum guaranteed runtimes, backfill, and proportional resource allocation to reduce GPU collection and node fragmentation. Operational tooling includes a simulator that replays traces, models fair-share scenarios and node variability at up to simulator 24x speed to vet scheduler changes. Resilience measures plus fault isolation cut downtime by 6x and interruptions by 5x while keeping utilization high.
AI-written summary · about
chapters (10)
- 0:00Large GPU cluster ops
- 1:36Scale journey and hardware
- 3:58Workload management overview
- 5:02Fair sharing for researchers
- 7:53Workload resilience and reliability
- 9:58Maximizing GPU utilization
- 12:51Node fragmentation and inefficiencies
- 15:03Safe scheduling policies
- 17:52Simulation-driven validation
- 19:14Future of AI compute
related talks
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
01h 15mTutorial: Build, Operate, and Use a Multi-Tenant AI Cluster Based Entirely on Open Source
28mShared Yet Isolated at Scale: Building Multi-Tenant Inference Platform on…
09mLightning Talk: Bringing Multi-Tenancy to GPU-Based Workloads
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
38mImproving GPU Utilization using Kubernetes
41mBeyond Uptime: Revolutionizing Fintech Reliability
31mAI, CERN, and the Quest for GPU Custody: How CERN Leverages DRA for Efficient GPU Sharing