Improving GPU Utilization using Kubernetes
summary
Google Kubernetes Engine adds two concrete GPU-sharing features: time sharing (context-switched slices, configurable e.g. up to 10 shared nvidia.com/gpu resources on a T4) and multi-instance GPU (MIG) (A100 hardware partitions, up to seven 1G/5GB slices). Nodes are auto-provisioned, labeled and tainted for scheduling, and Horizontal Pod Autoscaler + cluster autoscaler use per-container GPU metrics. Time sharing risks aggregate OOM if containers exceed total GPU memory; MIG provides isolation and predictable QoS. Both are generally available.
AI-written summary · about
chapters (12)
- 0:00Introduction to GKE
- 2:03GPU utilization challenges
- 5:04GPU sharing solutions
- 6:22Time-sharing explanation
- 8:39Time-sharing configuration
- 12:53Memory management risks
- 14:24Auto scaling on GKE
- 19:54Multi-instance GPU (MIG)
- 23:14MIG deployment
- 24:09Comparing MIG and time-sharing
- 29:14Key takeaways
- 30:06Q&A session
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs- 32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
26mState of the GPU(nion)
30mRunning AI Workloads in Containers and Kubernetes
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes