Optimization's Untold Gift to Learning: Implicit Regularization

on YouTube ↗01h 01m

" summary

Shows that plain optimization picks a particular solution from many global minima: gradient methods on underdetermined linear least squares converge to the minimum-norm solution; gradient flow on factorized matrix models empirically and conjecturally selects the minimum nuclear-norm completion if initialized near zero with tiny steps (proven when measurements commute); for separable logistic loss gradient descent diverges in norm but converges in direction to the maximum-margin direction with WT ≈ w_mm · log T and margin improving only as 1/log T. Experiments show Path-SGD and SGD/Adam produce different generalization, so implicit bias depends on optimizer.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 2:53Increasing the Network Size (Number of Hidden Units)
3. 6:52AdaBoost
4. 11:40The Path-Norm
5. 16:36Where is the Regularization?
6. 21:07SGD vs ADAM
7. 23:52Different optimization algorithm
8. 27:06Simple Example: Least Squares
9. 28:25Moving On: Matrix Least Squares
10. 29:29Factorized Matrix Problems (Linear Neural Nets)
11. 38:50Warm Up: Gradient Descent on W
12. 41:07GD on U, single observation (m=1)
13. 44:41What we can prove: commutative A
14. 46:31The Non-Commutative Case
15. 50:30Logistic Regression on Separable Data
16. 54:10How Fast is the Margin Maximized?
17. 57:59Other Objectives and Opt Methods
18. 59:14Different Asymptotics

" related talks

• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 01h 17mImplicit Regularization INati Srebro · Deep Learning Boot Camp 2019
• 11mOn Implicit Regularization in Deep LearningWei Hu · Meet the Fellows Welcome Event
• 01h 05mOptimization for Machine Learning IElad Hazan · Foundations of Machine Learning Boot Camp 2017
• 46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient DescentNadav Cohen · Frontiers of Deep Learning 2019
• 01h 04mOptimization for Machine Learning IIElad Hazan · Foundations of Machine Learning Boot Camp 2017
• 46mKernel and Deep Regimes in Overparameterized LearningSuriya Gunasekar · Frontiers of Deep Learning 2019
• 47mIs Minimax Optimization the Right Framework to Understand Learning in Minimax Games?Gauthier Gidel · Adversarial Approaches in Machine Learning 2022