Practical Quantization in Keras
summary
Quantization in Keras converts FP32 weights to lower-bit formats using Keras’s unified quantise API and KerasHub models, yielding measurable tradeoffs: weight storage drops about 4x smaller weights going FP32→INT8, compute work falls by ~16x, a Gemma 3 1B demo runs quantized with visible but small quality change, INT4 shows larger degradation, and PTQ vs GPTQ calibration cuts perplexity increases from ~34% down to single-digit percents while delivering 30% peak GPU memory reduction and practical latency gains. INT8 and INT4 supported, per‑channel quantization currently used
AI-written summary · about
chapters (12)
- 0:00Practical quantization intro
- 1:28Quantization for edge devices
- 4:40Model size and latency reduction
- 7:11Hardware acceleration
- 11:30Uniform fine quantization
- 21:04Post-training workflow
- 23:10Keras and Keras Hub
- 27:23Live quantization demo
- 32:37Keras quantization internals
- 35:47Performance benchmarks
- 39:19Summary and closing
- 40:03Q&A
related talks
40mUnlocking Local LLMs with Quantization
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
10mQuantization
26mLow-Precision Inference without Quality Loss...
14mOcto: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny
01h 14mDistributed and concurrent optimization for machine learning
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
16mDecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization