Octo: INT8 Training with Loss-aware Compensation and Backward Quantization for Tiny
summary
Octo implements end-to-end INT8 training using a new Loss-aware Compensation mechanism and Parameterized Range Clipping so both forward and backward dot products run in 8-bit with int32 accumulators. A small learnable compensation layer (alpha, mu, beta) uses elementwise broadcasting and bit shifts to cancel quantization error; PRC removes zero-point offsets in gradients. Fake-quantization baselines fail to converge, while Octo matches full-precision accuracy with a modest ~15% data-conversion overhead but shorter per-iteration time and substantial peak/average memory reduction on commercial edge AI chips.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:25Rise of On-device Learning
- 1:17Common Compression Methods
- 1:41The Workflow of DNN Training
- 2:37Bridge the Gap: Data Quantization
- 3:15Why We Need Quantization?
- 4:02Potential Gains
- 5:10Co-design of Network and Training Engine
- 5:37Our System: Octo
- 6:44Loss-aware Compensation
- 7:39Backward Quantization
- 8:39Evaluation Setup
- 9:04Convergence Results
- 9:50Ablation Study: Impact of LAC and PRC
- 10:53Image Processing Throughput
- 11:19Deep Insight of Feature Distribution Visualization of intermediate Feature Distribution
- 11:59System Overhead
- 12:56Conclusion
related talks
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
48mPractical Quantization in Keras
40mUnlocking Local LLMs with Quantization
10mQuantization
16mDecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
26mLow-Precision Inference without Quality Loss...
23mTowards Bit-Level Dominance Preserving Quantization of Neural Classifiers
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM