SHEPHERD: Serving DNNs in the Wild
summary
SHEPHERD is a model serving system that combines a two-level design planner that partitions hundreds-to-thousands of request streams into groups and an online Flex online algorithm that uses preemption and batch-aware prioritization to bound goodput loss versus offline optimal. Grouping reduces short-term CV so planners provision more accurately; Flex corrects schedules online. Implementation on a 12-GPU testbed and large emulations with a Microsoft trace mapped to 17 DNNs yields up to 18.1X higher goodput and 1.8X better utilization while scaling to hundreds of workers.
AI-written summary · about
chapters (12)
- 0:00Start
- 0:25Model serving systems: background
- 1:17Model serving systems: importance
- 1:33Model serving systems: decisions
- 1:59Model serving system: goals
- 5:56Challenges in achieving high goodput
- 7:27Problems of existing solutions
- 7:57Break utilization-scalability trade-off
- 8:11Group-level predictability
- 10:54FLEX: Preemption
- 11:59Testbed experiment
- 13:08Large-scale emulation
related talks
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
26mEfficiently serving LLMs at scale.
12mPreview: ML Serving
20mScheduler and Sharding Considerations for Network Efficiency
16mSuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
36mAccelerating High-Performance Machine Learning at Scale i...
12mEasy and affordable access to GPUs for AI/ML workloads