DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
System disaggregates prefill and decoding onto separate GPUs, assigns independent parallelism and resource plans for each phase, and places instances to cut key-value cache transfer; implemented placement algorithms and online scheduling and tested on 13B–175B models across chat, code completion, and summarization workloads. Results show prefill and decoding disaggregation yields up to 7.4× more requests or 12.6× tighter SLO versus state-of-the-art while meeting >90% attainment, and KV-cache transfer costs are negligible (under 0.1% latency, 95% <30 ms).
AI-written summary · about
chapters (6)
- 0:00Introduction to DistServe
- 3:13Problems with LLM serving
- 6:51Disaggregating prefill and decoding
- 9:44DistServe design and optimization
- 12:54Evaluation results
- 14:19Conclusion and future
related talks
14mScalable Solutions for Running Large Language Models
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
26mEfficiently serving LLMs at scale.
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
16mRevisiting Pipeline Parallelism for LLM Serving
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
18mGPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale