Implicit Regularization II
summary
Claims that optimization selects specific solutions: for underdetermined least squares gradient descent from zero yields the minimum norm solution; for exponential-tail losses (logistic/cross-entropy) gradient descent drives weights to infinity but the direction converges to the max margin (hard‑margin SVM) solution with margin gap shrinking as 1/log T while loss drops as 1/T. Matrix factorization converges (conjecturally) to min nuclear norm (proved for commuting measurements); initialization scale controls a phase change between kernel (alpha→∞) and sparse/bridge biases (alpha→0).
AI-written summary · about
chapters (11)
- 0:00Optimization fundamentals
- 1:44Implicit bias in least squares
- 6:51Mirror and natural gradient
- 11:54Steepest descent limitations
- 16:32Logistic loss implicit bias
- 26:14Steepest descent with logistic
- 32:41Generalization and stopping
- 42:33Deep matrix factorization
- 54:04Implicit bias in deep nets
- 1:03:36Kernel regime vs deep regime
- 1:12:38Initialization scale impact
related talks
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
46mKernel and Deep Regimes in Overparameterized Learning
29mImplicit Regularization in Nonconvex Statistical Estimation
11mOn Implicit Regularization in Deep Learning
42mA Primal-dual Analysis of Margin Maximization by Steepest Descent Methods
01h 05mOptimization for Machine Learning I
01h 01mRobustness, Stochastics, Uncertainty 2
13mNon-Parametric Convergence Rates for Plain Vanilla Stochastic Gradient Descent