Production GPU Cluster with K8s for AI and DL Workloads
summary
NVIDIA describes building and operating a production Kubernetes GPU cluster for AI, deep learning and HPC. Coverage includes automated provisioning tools for GPU nodes and CUDA driver upgrades, a custom batch-job scheduler, and production monitoring with health checks and telemetry. The talk also outlines gaps around GPU affinity, sharing and co-scheduling and updates on ongoing projects to address them.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:20Massively Growing Computing Needs
- 1:04GPU Acceleration and Trends
- 2:17Containers & GPU-accelerated Applications
- 3:39NGC Registry: GPU-accelerated Software Hub
- 5:27NGC: GPU-accelerated Containers for Al & HPC
- 7:30Nvidia Saturn-V: Internal GPU Clusters
- 9:23Cluster Users and Workloads
- 11:52Job Submission Flow
- 13:07Architecture Overview
- 14:18K8s and HPC/Batch Limitations
- 16:20Batch: Queues, Quotas and Fair share
- 18:35Batch: Handling long running jobs
- 22:13Multi-node Jobs with MPI in K8s
- 23:42GPU Telemetry and Efficiency
- 24:58Telemetry data in Grafana
- 25:12Telemetry: Utilization of a 8-GPU job
- 26:07GPU Health and Pre-flight checks
- 27:29GPU Affinity
- 29:30Hitless Transition and Batch Federation
- 30:46Hitless Rolling Upgrade Operator
- 31:42Misc: Long Standing K8s Issues
- 32:53Summary
- 33:41Ongoing Work
related talks
40mBuilding Custom GPU Clusters at Scale: Using Kubespray To Create High-Performance AI Infrastructure
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
37mBuilding Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I]
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
32mScaling Kubernetes Clusters for Generative Models: Managing GPU Resources for AI App...
38mKubernetes for AI Workloads – What Works and What… Doesn’t
51mDeploying Deep Learning models on Kubernetes clusters with GPU Enabled