Dynamic Scheduling for Large Language Model Serving
summary
Llumnix is a dynamic scheduler built on vLLM and Ray that implements near-zero-downtime live migration of KV caches to continuously reschedule requests across model instances. Implemented and evaluated on 16 A10 GPUs with LLaMA-family models, it reports up to 2.2× mean and 5.5× P99 improvements in first-token latency, 1.3× faster P99 per-token generation, and cuts preemption latency cost by 73%. Llumnix is open source and deployed at Alibaba.
AI-written summary · about
chapters (12)
- 0:00Introduction and motivation
- 1:26Unique characteristics of LLM serving
- 3:16Scheduling challenges in LLM serving
- 6:06Dynamic rescheduling with Lumx
- 8:17Live migration mechanism in Lumx
- 10:28Scalability and scheduling benefits
- 12:54Prototype and evaluation results
- 14:02Implementation and production challenges
- 17:14Compatibility with inference engines
- 19:23Prefill-decode disaggregation
- 23:00Conclusion and future work
- 23:44Q&A session
related talks
16mLlumnix: Dynamic Scheduling for Large Language Model Serving
40mScalable and Efficient LLM Serving With the VLLM Production Stack
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
15mResource Multiplexing in Tuning and Serving Large Language Models
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
33mllm-d: Kubernetes Native Distributed Inferencing
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone