SDE Approximations and Scaling Rules for Distributed Deep Learning
summary
Derives stochastic differential equation approximations for large‑batch adaptive optimizers (RMSprop and Adam) and proves a Square Root Scaling Rule that contradicts the usual linear rule: learning rate should scale with the square root of batch size to preserve noise. Analyzes local-update methods via a slow SDE and proposes a Quadratic Synchronization Rule setting sync period proportional to 1/(learning rate)^2 as LR decays. Empirical validation across vision and language tasks shows the rules improve training and generalization (Vision Transformer: ~7 hours saved and +1% accuracy).
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:09Distributed training challenges
- 3:37Large batch training limits
- 5:05Linear scaling rule & SDE approx
- 6:53Adaptive methods & scaling rules
- 8:41Beyond trajectory matching
- 9:28Local SGD benefits
- 11:16Quadratic sync rule
- 12:40Future directions
related talks
41mOptimization Algorithms for Deep Learning
01h 01mRobustness, Stochastics, Uncertainty 2
31mUnderstanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
31mLarge Scale Stochastic Training of Neural Networks
33mStochastic Optimization for Big Data Machine Learning Problems
58mFirst-Order Stochastic Optimization
11mData Efficient Methods for Reinforcement Learning
33mTradeoffs in Large Scale Learning: Statistical Accuracy vs. Numerical Precision