GPU Reservations: Maximizing Utilization and Fairness Across Teams
summary
Company pair builds a Kubernetes platform that combines reserved capacity with autoscaling to claim up to 75% cost savings while keeping researcher elasticity. It uses Run:AI / Nvidia K-scheduler for hierarchical queues, topology-aware placement, and transparent adaptive preemption guided by team priorities, plus Lightning.AI observability with queue metrics and preemption history. The fleet is ~40,000 GPUs at ~95% utilization, includes automated GPU fault detection/coordination with hardware providers, and can burst to on-demand when budgets require.
AI-written summary · about
chapters (7)
- 0:00GPU infrastructure challenges
- 3:10Lightning AI platform overview
- 7:02Kai scheduler architecture
- 10:11GPU error mitigation
- 14:26Future GPU scheduling needs
- 18:42Lightning AI roadmap
- 22:30Q&A
related talks
21mExperience Operating Large GPU Clusters at Organizational Scale
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
20mHow are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
38mImproving GPU Utilization using Kubernetes
41mBeyond Uptime: Revolutionizing Fintech Reliability
24mImproving GPU Utilization and Accelerating Model Training with Scheduling Framework and NRI
31mAI, CERN, and the Quest for GPU Custody: How CERN Leverages DRA for Efficient GPU Sharing