LLM Inference Optimization From Simple To Sophisticated
summary
Practical roadmap: choose a model just “smart enough” and quantize to get ~1.3–1.7x speedups, often delivering ~50% gains from model choice; use vLLM, SGLang, TensorRT-LLM for continuous batching and KV-cache management; route by load+KV-awareness and consider KV offloading to host memory or SSDs; distribute with tensor or pipeline parallelism (works across Ethernet) and run at least three replicas for production; advanced moves include speculative decoding with a 0.5–1B draft model and disaggregated prefill/decode for >100B models. Metrics: time-to-first-token, inter-token latency, throughput, cost-per-token.
AI-written summary · about
chapters (13)
- 0:00LLM inference intro
- 1:24Self-host LLM?
- 2:52Optimization goals
- 4:35Inference stack layers
- 6:00Level one: model choices
- 9:30Routing and quantization
- 10:54Level two: frameworks
- 13:20KV cache management
- 15:09Parallelism strategies
- 17:16Custom optimizations
- 20:07Disaggregated inference
- 21:35Fine-tune smaller models
- 23:01Self-hosting approach
related talks
30mLLM Inference Arithmetics: the Theory behind Model Serving
30mLLMs: Beyond the Hype - A Practical Journey to Scale
30mAccelerated LLM Inference with Anyscale
38mOptimizing LLM Inference for the Rest of Us
21mScaling LLM Inference with vLLM
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone