Efficiently serving LLMs at scale.
summary
Deploys a 176B Bloom (≈350 GB of weights) across multi-GPU nodes using DeepSpeed tensor parallelism, a Rust front end with Python shards from Hugging Face text-generation-inference, and an open-source Model Mesh lineage to turn costly single-user inference into multi-tenant throughput. Key claims: continuous batching and KV cache concatenation let the server add and prune sequences mid-decode, dynamic batch sizing supports variable lengths while keeping GPU memory bounded, flash attention and vectorized decoding cut padding and GPU ops, and VM-style noncontiguous KV storage reduces copies — combined work produces much higher utilization and batches up to ~128–256.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 3:25Early LLM deployment challenges
- 7:16Continuous batching explained
- 10:45KV cache optimization techniques
- 13:35Dynamic batch sizing
- 15:22Flash attention and memory savings
- 17:08Vectorized decoding improvements
- 18:34Dynamic split-fuse technique
- 20:21Virtual memory for KV cache
- 21:27Speculative decoding with small models
- 23:10Emerging inference servers
- 24:14Additional optimization methods
related talks
36mAccelerating High-Performance Machine Learning at Scale i...
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
01h 18mOptimizing LLM Training and Inference Performance on GPUs
10mvLLM & Deepspeed Updates
19mNo GPU Left Behind: Scaling Online LLM Training With...
15mResource Multiplexing in Tuning and Serving Large Language Models
31mThe Evolution of Multi-GPU Inference in vLLM
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...