The Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
summary
Baseten’s deck claims concrete wins: FP8 quantization on Lovelace/Hopper/Blackwell cuts bandwidth with preserved quality, distilling DeepSeek R1 671B into ~70B yields ~10x resource reduction, and TensorRT LLM / vLLM / ggml-style frameworks beat vanilla transformers. Benchmarks show batching tradeoffs (1–128) and speculative methods that produce multiple tokens per forward pass raise latency/throughput efficiency, while KV cache reuse slashes time-to-first-token for long contexts and FP8 preserves dynamic range compared with int8 quant. Multi-node B200 setups scale but can cost hundreds $/hr.
AI-written summary · about
chapters (12)
- 0:00Inference optimization intro
- 1:03GPUs and fast inference
- 6:59Model selection and distillation
- 9:26Quantization and FP8 benefits
- 10:50Batch inference trade-offs
- 12:17Disaggregated serving
- 13:38Long context handling
- 14:41Speculative decoding
- 17:07KV cache reuse
- 19:10Multi-GPU parallelism
- 22:22Summary and insights
- 23:15Q&A: dedicated vs shared
related talks
26mAmazingly Fast and Incredibly Scalable Inference...
23mThe Race to the Bottom: Low Latency in the age of the Transformer
26mLow-Precision Inference without Quality Loss...
31mLow-Latency Solutions for Resource-Constrained Environments
01h 18mOptimizing LLM Training and Inference Performance on GPUs
36mHigh-Performance Distributed Tensorflow
24mLLM Inference Optimization From Simple To Sophisticated
38mOptimizing LLM Inference for the Rest of Us