Hitting the High-D(imensional) Notes: An ODE for SGD learning dynamics in high-dimensions

on YouTube ↗56m

" summary

Framework shows that when samples and dimensions grow together, SGD and SGD+M trajectories become an ODE limit determined only by the data covariance spectrum; proof uses random matrix resolvents, homogenization of the SGD noise, and Ito calculus to reduce dynamics to a Volterra/ODE for least squares. Main quantitative claims: explicit learning-rate and momentum formulas in terms of Hessian spectra, a sharp implicit conditioning ratio (ICR) threshold for batch-size beyond which SGD+M attains full-batch acceleration, and empirical matches on real datasets.

AI-written summary · about

" chapters (11)

1. 0:00Introduction and motivation
2. 2:58Predicting SGD behavior
3. 7:00High-dimensional structure
4. 11:26Theoretical setup
5. 15:10Concentration and statistics
6. 18:20Resolvent and time scaling
7. 22:11Stochastic process analysis
8. 28:40Deterministic equivalents
9. 34:56Multipath and delocalization
10. 44:36Volterra equations and mini-batching
11. 50:28Conclusion and Q&A