Maximizing GPU Utilization for Generative AI
summary
How to maximize GPU utilization for generative AI workloads running on OpenStack and Kubernetes by selecting and configuring GPU-sharing and isolation technologies. It covers NVIDIA vGPU, CUDA streams, time-slicing, CUDA Multi-Process Service, NVIDIA MIG, and AMD SR-IOV, and gives practical configuration and optimization advice for inference and training workloads to improve utilization and lower operational cost.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:22GPU needs for generative AI
- 10:05GPU use cases by workload
- 12:14GPU sharing tech
- 23:20Time slicing and overprovisioning
- 25:53Multiple Instance GPU (MIG)
- 32:40NVIDIA MPS for logical sharing
- 36:11Dynamic resource allocation
- 40:09GPU monitoring and optimization
- 43:21Q&A
related talks
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
20mHow are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
32mScaling AI Workloads with Kubernetes: Sharing GPU Resources Across Multiple Containers
34mGPU as a Service Over K8s: Drive Productivity and Increase Utilization
37mCo-Location of CPU and GPU Workloads with High Resource Efficiency