AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
summary
AlpaServe uses model parallelism not just to fit big models but to statistically multiplex GPUs across many models, trading partition overhead for burst tolerance. It generates inter- and intra-operator partition plans, runs a simulator-guided greedy two-stage placement, and deploys on a 64-GPU AWS cluster serving Transformer and vision models from a Microsoft production trace. Results show up to 10× higher request rate or 6× more burstiness while meeting latency SLOs for >99% of requests.
AI-written summary · about
chapters (8)
- 0:00AlpaServe intro
- 1:34Model parallelism for serving
- 3:44Statistical multiplexing
- 7:14Real-world cases
- 10:25Overheads of strategies
- 12:10AlphaServe design
- 13:56Experimental results
- 14:38Summary
related talks
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
Power-aware Deep Learning Model Serving with μ-Serve
16mSuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
40mScalable machine-learned model serving
26mEfficiently serving LLMs at scale.
36mAccelerating High-Performance Machine Learning at Scale i...
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
12mPreview: ML Serving