On Implicit Regularization in Deep Learning

on YouTube ↗11m

" summary

Argues that gradient-descent optimization imposes an implicit regularizer that prefers low-complexity solutions, shows for matrix completion that parametrizing M as a product of L D×D matrices plus near-zero initialization recovers a low-rank ground truth, derives dynamics of singular values showing a depth-dependent rich-get-richer effect, and proves a two-layer ReLU network trained by gradient descent computes an almost linear function during early training, with experiments matching theory.

AI-written summary · about

" chapters (11)

1. 0:00Intro
2. 0:13Deep Learning Pipeline
3. 1:07Over-parameterized Neural Nets Can Generalize Well
4. 2:32Implicit Regularization
5. 3:16Matrix Completion
6. 6:22Analyzing the Dynamics of GD
7. 7:08GD Prefers Low-Complexity Solutions
8. 8:00Neural Network Learns Functions of increasing Complexity
9. 8:52Main Result
10. 9:50Experiments
11. 10:05Takeaways

" related talks

• 29mImplicit Regularization in Nonconvex Statistical EstimationYuxin Chen · Optimization, Statistics and Uncertainty
• 01h 17mImplicit Regularization INati Srebro · Deep Learning Boot Camp 2019
• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient DescentNadav Cohen · Frontiers of Deep Learning 2019
• 38mWhy Deep Learning Works: Implicit Self-Regularization in Deep Neural NetworksMichael Mahoney · Randomized Numerical Linear Algebra and Applications 2018
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 13mThe dynamical view of machine learning: implicit biases of large learning ratesYuqing Wang · Meet the Fellows Welcome Event Fall 2024