FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
summary
FastServe implements iteration-level preemption for autoregressive LLMs, letting the system preempt after each token instead of running to completion. It combines a skip-join multi-level feedback queue that uses known input length to pick an initial priority and avoid unnecessary demotions, with priority KV-cache swapping that proactively offloads/uploads KV state between GPU and host to hide PCIe latency. Deployed as a distributed serving runtime, it shows up to 6.1× throughput vs vLLM and 31–74× vs FastTransformer under real workloads.
AI-written summary · about
chapters (10)
- 0:00FastServe introduction
- 1:42LLM inference phases
- 2:46Head-of-line blocking
- 4:09Iteration-level preemption
- 5:54Skip Job queue design
- 8:23GPU memory overhead
- 9:47KV cache swapping
- 10:29Evaluation and throughput
- 12:18Fairness in memory
- 13:21Conclusion
related talks
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
15mResource Multiplexing in Tuning and Serving Large Language Models
14mScalable Solutions for Running Large Language Models
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
30mLLM Inference Arithmetics: the Theory behind Model Serving
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving