A Deep Dive on Supporting Multi-Instance GPUs in Containers and Kubernetes
summary
Demonstrates concrete MIG support for containers and Kubernetes using the nvidia container toolkit, nvidia k8s-device-plugin, and nvidia-mig-parted. Converts an eight‑GPU DGX A100 (8×40GB) into 56 1g.5GB MIG devices, with mode switching requiring a full GPU reset that takes ~5 minutes and VM reboots in cloud VMs. Creation order matters to avoid fragmentation; each MIG device injects three device nodes and limited /proc views. A systemd wrapper persists partitions across reboots and the demo shows pods successfully consuming MIG devices.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:34GPUS AND KUBERNETES Seamlessly scale up training and inference to a cluster of GPU machines
- 2:20WHAT ARE MULTI-INSTANCE GPUs? Slices of a full GPU with dedicated memory and compute resources
- 3:32OUTLINE
- 4:15MULTI-INSTANCE GPUs (MIG)
- 12:10GPUS AND CONTAINERS The NVIDIA Container Toolkit
- 14:30GPUS AND KUBERNETES Allocate GPUs to pods in a Kubernetes Cluster
- 15:40MIG IN CONTAINERS AND KUBERNETES
- 17:34SYSTEM LEVEL INTERFACE FOR MIG
- 19:59CHALLENGES WITH MIG PARTITIONING How do I create a MG Device in the first place?
- 22:40MIG PARTITION EDITOR
- 31:42SUMMARY AND CONCLUSION
related talks
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers- 32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
38mImproving GPU Utilization using Kubernetes
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
30mRunning AI Workloads in Containers and Kubernetes
39m✨ GPU in My Containers!
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs