Scaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
summary
Kubernetes uses the nvidia GPU operator plus the device plugin to label nodes, install drivers and expose GPU allocatable resources; MIG physically shards A100/H100 into up to seven compute slices (eight memory slices) so a single A100 can advertise seven 1G devices; alternative is Kubernetes-managed time-slicing replicas that round-robin processes and expose many logical devices but sacrifice SMP and memory isolation; GPU operator’s MIG manager reads saved profiles to partition cards and node selectors/taints ensure GPU nodes host only GPU workloads.
AI-written summary · about
chapters (13)
- 0:00Scaling AI workloads
- 3:08Why Kubernetes for GPUs
- 4:11Multi-instance GPU (MIG)
- 7:19Partitioning GPU slices
- 11:36NVIDIA GPU Operator
- 14:26Dedicated GPU nodes
- 16:13GPU sharing limits
- 17:56Sharing GPUs via MIG
- 21:05Time slicing GPUs
- 23:35MIG vs time slicing
- 25:44Optimizing deep learning
- 28:12Summary
- 28:33Question and answer session
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
30mRunning AI Workloads in Containers and Kubernetes
38mImproving GPU Utilization using Kubernetes
05mLightning Talk: Sharing a GPU Among Multiple Containers
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
39m✨ GPU in My Containers!