LLM Inference Arithmetics: the Theory behind Model Serving
summary
Argues that transformer inference is inherently serial while training is parallel, so training needs ~3× the FLOPs of inference and attention compute scales quadratically with sequence length (T²). Shows KV cache trades compute for memory — KV cache ≈ 20% of model size (e.g., Mistral 7B ≈ 20% extra VRAM) and splits generation into a fast prefill phase and a slow single-token decode phase. Benchmarks and engine notes compare TensorRT (high-performance, Nvidia-tuned), VLM/ggml and llama.cpp for practical self‑hosting tradeoffs.
AI-written summary · about
chapters (13)
- 0:00Welcome
- 1:15Talk overview
- 3:21Training vs inference
- 4:47Model requirements
- 6:35Compute and memory
- 9:44Memory in training
- 11:28Transformer architecture
- 13:58Attention complexity
- 16:49KV cache impact
- 20:22Prefill and decoding
- 23:10Batching strategies
- 24:13Self-hosting viability
- 25:37Q&A
related talks
24mLLM Inference Optimization From Simple To Sophisticated
34mLLM Inference Arithmetics
34mOptimizing AI Inference with ML Compilers & Hardware
41mSub-Linear Algorithms meets Large Language Models
30mAccelerated LLM Inference with Anyscale
24mCost and Performance Optimization of LLM Inferencing.
26mEfficiently serving LLMs at scale.
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...