How are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
summary
Run:ai on Azure Kubernetes Service claims to raise GPU utilization from typical ~20% to much higher by adding an AI workload scheduler with project queues, guaranteed quotas plus over‑quota provisioning, automated preemption with checkpoints, and a fairness algorithm; it supports Fractional GPU allocation and on‑demand NVIDIA MIG partitions so multiple containers share one GPU, and provides self‑service Workspaces (Jupyter, MLflow, W&B, VS Code) with data mounting and marketplace deployment.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 5:13Why are GPUs underutilized
- 6:32How do we provision GPUs
- 7:57Dynamic GPU provisioning
- 9:06Our platform
- 11:57Automation of resource allocation
- 15:35Workspaces
related talks
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
21mUnlocking How To Efficiently, Flexibly, Manage and Schedule Seven AI Chips in Kubernetes
50mMaximizing GPU Utilization for Generative AI
38mImproving GPU Utilization using Kubernetes
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency