Towards GPU Utilization Prediction for Cloud Deep Learning
summary
A prediction engine extracts features from each model computation graph (13 base CNN architectures permuted to 81 workloads) and computes FLOPs, tensor shapes and node counts, then predicts GPU utilization with a random forest regressor (RMSLE 0.154). Experiments use NVIDIA 1080/2080 hardware and tools like nvidia-smi and Nsight, and a modified Kubernetes scheduler running first-fit-decreasing collocation, yielding ~69% average cluster GPU utilization versus 43.1% slot-based.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:29Deep Learning (DL) Systems
- 1:23DL System Challenges
- 2:08Online profiling approach
- 2:41DL Metrics
- 3:53Case: Scheduling
- 4:26Time is Money
- 6:20Objective
- 6:58DL computation graph
- 7:42Analysis
- 10:47GPU Utilization Prediction
- 12:16Case Study: DL Cluster Scheduler
- 13:15Evaluation
- 13:57Open Challenges
- 15:08Conclusions
related talks
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
35mUtilization Is the Key to Efficiency: What It Takes to Run...
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
34mGPU as a Service Over K8s: Drive Productivity and Increase Utilization
20mHow are You Utilizing GPUs? Best Practices on Managing GPUs in Azure
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
32mGPU Machine Learning From Laptop to Cloud