Implicit Regularization I

on YouTube ↗01h 17m

" summary

Overparameterized nets and matrix completion have many global minima, yet the chosen optimizer determines which beach the solution lands on: SGD vs Adam difference shows both reach zero training loss but give different test error; factorizing W = U V (two n×n factors, 2n^2 parameters) and running gradient descent biases toward the min-nuclear-norm solution, recovering low‑rank matrices on held‑out entries; minibatch SGD and step‑size choices (exact line search vs small fixed step) change the implicit regularizer and thus generalization.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 3:10Boosting
3. 4:44Complexity Control
4. 17:02Optimization Landscape
5. 19:24Biases
6. 20:09Matrix Completion
7. 22:54Gradient Descent
8. 27:00Outline
9. 28:23Goal of Learning
10. 30:17Example
11. 35:48Stochastic Optimization
12. 38:15Recap
13. 41:52Stochastic Gradient Descent

" related talks

• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 11mOn Implicit Regularization in Deep LearningWei Hu · Meet the Fellows Welcome Event
• 23mOn Implicit Regularization in Deep LearningWei Hu · Meet the Fellows Welcome Event
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 29mImplicit Regularization in Nonconvex Statistical EstimationYuxin Chen · Optimization, Statistics and Uncertainty
• 01h 37mTutorial: Implicit Bias IINati Srebro · Deep Learning Theory Workshop and Summer School 2022
• 47mIs Minimax Optimization the Right Framework to Understand Learning in Minimax Games?Gauthier Gidel · Adversarial Approaches in Machine Learning 2022
• 01h 04mOptimization for Machine Learning IIElad Hazan · Foundations of Machine Learning Boot Camp 2017