Databricks' vLLM Optimization for Cost-Effective LLM Inference
summary
Databricks reports concrete vLLM fixes: profiling shows GPU idle time climbs from ~2.6% at batch 1 to ~25% at large batches for 70B models, and release 0.6.0+ yields a measured 2.7x throughput and 5x latency improvement. Engineering changes include asynchronous output processing (reduces GPU idle time, giving ~9–14% speedup at high RPS) and multi-step scheduling (amortizes CPU cost; combined with async gives ~1.23x end-to-end gain). FP8 quantization unlocks higher concurrency with no major quality loss; speculative decoding (4–8 token lookahead) gives ~2–2.5x speedups at low batch sizes with >70% acceptance.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 1:50Databricks LLM serving products
- 3:17Why continue using vLLM
- 5:46vLLM 0.6.0 performance upgrades
- 7:09Asynchronous output processing
- 11:43Multi-step scheduling
- 15:57Other optimizations FP8 and speculative decoding
- 18:29Conclusion and future work
- 19:34Q&A session
related talks
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
24mLLM Inference Optimization From Simple To Sophisticated
21mScaling LLM Inference with vLLM
38mOptimizing LLM Inference for the Rest of Us
24mCost and Performance Optimization of LLM Inferencing.
08mHacks to Make LLM Training Faster
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
16mHow fast are LLM inference engines anyway?