On Implicit Regularization in Deep Learning

on YouTube ↗23m

" summary

Shows that for matrix completion parameterized as a product of d×d layers, gradient descent from near-zero consistently recovers the low-rank ground truth by driving singular values with a rich-get-richer among singular values dynamic so rank grows gradually and low-rank minima are reached. Also proves for a two-layer nonlinear net that early training closely matches a linear model trained by gradient descent, with experiments confirming the coupling and the initial linearization.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 2:18Learning and Games
3. 4:46Decisions
4. 12:07Deep Learning Pipeline
5. 13:02Generalization Mystery
6. 14:27Implicit Regularization
7. 15:47Deep Learning Approach
8. 20:47Formal Results

" related talks

• 01h 17mImplicit Regularization INati Srebro · Deep Learning Boot Camp 2019
• 38mWhy Deep Learning Works: Implicit Self-Regularization in Deep Neural NetworksMichael Mahoney · Randomized Numerical Linear Algebra and Applications 2018
• 29mImplicit Regularization in Nonconvex Statistical EstimationYuxin Chen · Optimization, Statistics and Uncertainty
• 46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient DescentNadav Cohen · Frontiers of Deep Learning 2019
• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 13mThe dynamical view of machine learning: implicit biases of large learning ratesYuqing Wang · Meet the Fellows Welcome Event Fall 2024