The Path to GPU as a Service in Kubernetes
summary
NVIDIA outlines getting GPUs into Kubernetes production: a year-long effort that moves from 1.6 limits (one GPU per node) to a device plugin subsystem in 1.8 that registers GPUs via a daemonset, exposes health checks and metrics, and supports the NVIDIA container runtime across OCI runtimes. The talk demonstrates zero-downtime registration via gRPC sockets, quantifies node startup delays (5–15 minutes on cloud), and explores practical fixes (pre-provision nodes, separate data pods using IPC) plus work on GPU sharing and topology and richer monitoring.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:05RUNNING A GPU APPLICATION
- 9:57GPU Pod
- 10:34Updating for GPU Services
- 11:47Resiliency for GPU Services
- 14:08Scaling for GPU Services
- 15:01Operating a GPU Cluster
- 17:34Device Plugin Lifecycle
- 19:38Zero Downtime Registration
- 21:50GPU Monitoring
- 22:44Current Challenges
- 25:59Technical Challenges Ahead
related talks
32mGPU Machine Learning From Laptop to Cloud
38mBeyond Kubernetes: Adapting To Spe...
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
21mExperience Operating Large GPU Clusters at Organizational Scale
31mAI, CERN, and the Quest for GPU Custody: How CERN Leverages DRA for Efficient GPU Sharing
18mArchitecting Multi-tenant Data-center Networks for GPU Customers
11mGPU‑Accelerated Workloads on KubeVirt: Scaling ML/AI in Kuberne...
06mAutoscaling GPU Clusters Anywhere — Hyperscalers, Neoclouds & Ba...