Quantization
summary
PyTorch 1.3 adds eager-mode 8-bit quantization API with three workflows: dynamic one-line quantization (weights → int8, activations float), post-training quantization using observers to calibrate ranges, and quantization-aware fine-tuning. Typical CNNs convert with under 0.5% accuracy loss and run ~2x faster on server CPUs; MobileNet after fine-tuning runs ~4x faster on ARM. Models export via torchscript, support operator implementations on FBGEMM and QNNPACK backends, and allow manual fusion/observer tweaks.
AI-written summary · about
chapters (10)
- 0:00Introduction to Quantization
- 0:52Quantization Fundamentals
- 1:55PyTorch Quantization Support
- 2:32Quantization Workflows
- 3:40Implementation Details
- 4:42Calibration and Conversion
- 5:58API and Future Features
- 7:11Performance Results
- 8:24Backends and Expansion
- 8:54Conclusion and Resources
related talks
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
35mClient Side Deep Learning Optimization with PyTorch
48mPractical Quantization in Keras
14mOptimize your TensorFlow Lite models
40mUnlocking Local LLMs with Quantization
01h 00mServing Pytorch Models in Production
01h 14mDistributed and concurrent optimization for machine learning
16mtorchtune: Easy and Accessible Finetuning in Native PyTorch