Unlocking Local LLMs with Quantization
summary
Quantization reduces model size roughly 4× (FP32→INT8) and shows near‑zero accuracy loss for 8‑bit while 4‑bit begins to degrade; experiments report 1.5× inference speedup with FP8 vs BF16 on H100, and a community toolchain can quantize a 70B model in about a minute. The workflow includes post‑training schemes (bitsandbytes, GPTQ, GGUF/llama.cpp), QAT and a published 1.58‑bit ternary model, and fine‑tuning via PEFT + QLoRA with merged adapters that maintain accuracy.
AI-written summary · about
chapters (14)
- 0:00Quantization and local LLMs
- 1:55Model compression overview
- 5:07What is quantization
- 6:41Linear quantization step-by-step
- 10:21Quantization advantages
- 12:29Quantization in development
- 14:39Quantization-aware training
- 19:04Post-training quantization
- 22:38Running quantized models
- 24:03LoRA fine-tuning quantized
- 27:52Industry adoption future
- 28:59Q&A part one
- 32:59Q&A part two
- 37:43Final Q&A closing
related talks
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
48mPractical Quantization in Keras
29mRunning Large Language Models on Laptops...
38mOptimizing vLLM Performance through Quantization
26mLow-Precision Inference without Quality Loss...
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models