PPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via...
summary
PPipe implements pool-based pipeline parallelism to split CNN models across heterogeneous GPUs, uses an MILP-based control plane to pick partition points and GPU counts, and runs a resource reservation-based adaptive batching data plane that picks batch size and path to meet latency SLOs. Evaluated on 18 CNNs with a 25H/75L GPU simulator and a 16-GPU Google Cloud deployment, PPipe raises low-class GPU utilization 41.1%–65.5%, boosts overall throughput 32.2%–75.1%, and sustains higher 99th-percentile SLO load (32%–48% more) than baselines.
AI-written summary · about
chapters (13)
- 0:00PPipe introduction
- 1:01Video analytics challenges
- 2:05Heterogeneous GPU clusters and SLO constraints
- 3:10Naive pipeline pitfalls
- 4:58Latency diversity across GPUs
- 5:38Pipeline parallelism limits
- 7:27Pool-based pipeline parallelism design
- 8:49PPipe problem formulation
- 11:16Handling online delays
- 13:22Evaluation setup
- 14:27Performance results
- 15:49GPU utilization insights
- 16:10Conclusion
related talks
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
16mRevisiting Pipeline Parallelism for LLM Serving
21mEnvPipe: Performance-preserving DNN Training Framework for Saving Energy
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
15mPipeThreader: Software-Defined Pipelining for Efficient DNN Execution
21mMulti Runtime Serving Pipelines
23mHeterogeneous Pipelines as the Unlock for ML at Scale