Scalable and Efficient LLM Serving With the VLLM Production Stack
summary
Open source vLLM Production Stack combines vLLM Production Stack router, an LMCache KV cache backend, Kubernetes + Helm deployment and SkyPilot for cross‑cloud GPU provisioning to cut latency and cost. Benchmarks on Llama 3 18B show time‑to‑first‑token falls from ~8s to ~1.4s when enabling GPU direct RDMA and KV cache sharing, with kernel‑optimized CPU↔GPU store/load, KV cache compression and prefix‑aware routing. The system is packaged as a two‑click deployable stack and runs successfully in production‑style demos.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 1:11Challenges in LLM serving
- 10:33LMCache architecture and KV cache
- 12:48Technical optimizations and RDMA
- 19:35Disaggregated PD demonstration
- 25:42LMCache cloud deployment
- 31:59Q&A and concluding remarks
related talks
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
26mLMCache: Lower LLM Performance Costs in the Enterprise
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
45mEverything You Need to Know About Running LLMs Locally
42mStop Treating LLMs Like REST APIs