The GPU Utilization Problem: What’s Going Wrong and How to Solve It
summary
Claim: typical data-center GPUs sit at 20–30% utilization now while the goal is 80–90%. Root causes: coarse Kubernetes GPU allocation (nvidia.com/GPU1), fragmented gang-scheduled training, noisy neighbors, and lack of visibility. Concrete fixes and tools: enable DRA, gang scheduling and Kueue/Volcano/Kai/Slurm, use MIG (example: A100 40GB → seven-way slices or vGPU fractions), implement queueing, bin-packing, checkpointing and preempt/resume. The team builds a GPU-as-a-Service combining these layers and demonstrates checkpoint + resume working for preempted jobs.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:47Agenda and problem overview
- 2:28GPU utilization in AI/ML
- 3:32Kubernetes GPU limits
- 5:18Dynamic resource allocation
- 8:30GPU job scheduling mistakes
- 10:59Gang scheduling necessity
- 13:48One GPU per workload myth
- 16:13Why multiple solutions must work together
- 21:34Multi-Instance GPU solutions
- 23:43GPU layer utilization issues
- 25:06Tools for GPU utilization
- 27:12GPU as a Service framework
- 30:41Demo: checkpointing and preemption
- 34:19Hardware vs software partitioning
- 37:54Workload prioritization
- 40:00Latency solutions
- 42:16Recommended tools
- 42:39Q&A
related talks
38mImproving GPU Utilization using Kubernetes
20mHow are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
32mAre You Really Out of GPUs? How to Better Understand Your GPU...
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
21mExperience Operating Large GPU Clusters at Organizational Scale
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency