Distributed and concurrent optimization for machine learning
summary
Shows concrete compression-and-sync tradeoffs: QSGD 4-bit quantization cuts communicated bits ~8–10x with a theoretical ~2x slowdown in convergence but preserves accuracy in experiments (AlexNet 2.5x end-to-end speedup on 16 GPUs, VGG19 3.5x). A top-k sparsification at 0.5% plus 8-bit quantization scales a production speech model from 14 days on 16 GPUs to 1.7 days on 128 GPUs. Implements a runtime called ScaleML sparse/quantized allreduce and proves convergence for the sparsification scheme.
AI-written summary · about
chapters (6)
- 0:00Start
- 20:21neral Setting
- 41:38Properties
- 43:47Compression
- 50:11ry: Quantization
- 59:18A Scalable Communication Framework for ML
related talks
34mBeyond the Continuum: The Importance of Quantization in Deep Learning
40mUnlocking Local LLMs with Quantization
17mZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
16mDecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
28mNext-generation frameworks for Large-scale Machine Learning
36mAccelerating High-Performance Machine Learning at Scale i...
29mRunning Large Language Models on Laptops...