JITServe: SLO-aware LLM Serving with Imprecise Request Information
summary
JITServe is an LLM-serving system that schedules with noisy length and dependency estimates, refines them online with quantile regression forests and pattern-graph clustering, and allocates just-enough bandwidth per time frame using a grouped-margin GMAX optimizer that forms homogeneous batches to boost efficiency. The paper builds a working 16-GPU prototype across dense and MoE models, reports 1.4×–6.3× goodput improvement, achieves 28.5%–83.2% resource savings, and runs within 3–9% of an oracle.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:42Heterogeneous SLO requirements
- 3:09LLM request types and SLOs
- 4:34Scheduling challenges with imprecise info
- 6:01JITServe system components
- 7:26Handling length uncertainty with QRF
- 7:47Resolving dependency uncertainty
- 8:51GMAX scheduling algorithm
- 10:14Two-step scheduling for throughput
- 11:17Evaluation setup and baselines
- 11:59Performance under increasing load
- 12:41Near-Oracle performance
- 13:03Conclusion
related talks
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
34mImprove AI Inference (serving models) With KServe and VLLM
28mSponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
26mEfficiently serving LLMs at scale.
30mLLMs: Beyond the Hype - A Practical Journey to Scale
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes