Zico: Efficient GPU Memory Sharing for Concurrent DNN Training
summary
Zico builds a shared GPU memory pool plus a memory manager that uses CUDA events and an in‑flight kernel counter to safely reclaim and limit allocations, and a scheduler called Scrooge that predicts per‑job allocation/deallocation patterns to overlap allocation phases across jobs. Implemented on TensorFlow and evaluated on two GPUs with six benchmarks, it achieves up to 1.6× throughput versus temporal sharing and up to ~4.7–7.5× versus NVIDIA MPS in oversubscription cases, while dynamically packing iterations to stay within GPU capacity.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:32GPU Utilization in DNN Training
- 1:27Existing GPU Sharing Solution
- 2:43Memory Sharing Opportunity
- 3:09Zico Overview
- 3:39Contributions
- 4:21Asynchrony between CPU and GPU
- 4:49Synchronization between CPU and GPU
- 5:29Early Memory Allocation
- 6:10Controlling Inflight Kernel
- 6:24Roadmap
- 7:17Limitation of Naïve Scheduling
- 7:59Scrooge Scheduler
- 9:59Evaluation
- 10:33Throughput: Identical Jobs
- 11:22Throughput: Identical Models
- 12:24Throughput: Non-identical Jobs
- 14:10Scheduling Example
- 15:12Summary
related talks
15mTransparent GPU Sharing in Container Clouds for Deep Learning Workloads
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
15mColocating ML Inference and Training with Fast GPU Memory Handover
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
38mMinimizing GPU Cost for Your Deep Learning on Kubernetes
44mGPU Sharing for Machine Learning Workload on Kubernetes
27mPyTorch Symmetric Memory: A New Programming Paradigm for Distributed AI
08mLightning Talk: GPU.x - Dynamic GPU Fraction and Sharing on ML Mainstream Framework