Optimizing vLLM Performance through Quantization
summary
vLLM adds practical quantization paths — activation quantization (FP8/INT8) and weight-only (W4/W8) — plus conversion kernels Marlin and Machete and a repo of pre-quantized checkpoints on Hugging Face. Using vLLM plus llm-compressor compresses activations and weights so linear layers run up to ~2x faster on Hopper/H100 tensor cores, often cutting a 70B deployment from four GPUs to two while preserving accuracy via fine-grain per-channel and per-token scaling.
AI-written summary · about
chapters (8)
- 0:00Introduction to vLLM
- 1:06Quantization fundamentals
- 5:26Quantization support in vLLM
- 11:44The case for activation quantization
- 19:16Performance analysis
- 23:11Accuracy and evaluation
- 27:31Roadmap and future outlook
- 30:04Q&A and technical discussion
related talks
21mScaling LLM Inference with vLLM
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
16mDecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
40mUnlocking Local LLMs with Quantization
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
24mLLM Inference Optimization From Simple To Sophisticated