Tutorial: Implicit Bias II

on YouTube ↗01h 37m

" summary

Experiments fit MNIST nets with 8,16,64,512,1000+ hidden units and show zero training error on random labels (Zhang et al.), yet test error keeps falling; kernel SVM and 1‑NN give analogous behavior. Optimization, not the class size, creates the bias: gradient descent converges to a minimum L2‑norm/max‑margin direction while coordinate descent yields an L1‑style bias. Matrix‑completion (U×V) trained by GD attains low test error (≈10% → ≈5% with different GD variants). Infinite‑width ReLU limits minimize ∫|f''| and benign overfitting arises under specific covariance spectral conditions.

AI-written summary · about

" chapters (34)

1. 0:00Introduction and experiment
2. 1:06Classic theory vs practice
3. 2:51The generalization mystery
4. 5:07Shattering and generalization
5. 7:20Implicit bias in algorithms
6. 9:35Complexity and simplicity
7. 12:15Finding generalizable minima
8. 14:17Complexity measures
9. 17:18Least squares bias
10. 19:42Optimization and simplicity
11. 22:55Logistic loss divergence
12. 25:04Algorithm-induced bias
13. 26:52Regularization and time
14. 29:15Infinite network concept
15. 33:45Matrix completion example
16. 38:10Numerical stability
17. 42:25Linear networks
18. 46:11ReLU networks
19. 49:22Architecture and geometry
20. 52:01Complexity approach
21. 55:28Approximation limitations
22. 57:36Matrix factorization rank
23. 1:02:14Theory vs phenomena
24. 1:04:26Overfitting definition
25. 1:06:54Double descent
26. 1:10:16Benign overfitting
27. 1:13:54Characterizing overfitting
28. 1:15:35Uniform convergence limits
29. 1:18:09Linear regression conditions
30. 1:20:39Pareto frontier
31. 1:23:14Summary and future work
32. 1:25:33Uniform convergence debate
33. 1:27:49Mathematics vs engineering
34. 1:29:18Questions and answers

" related talks

• 46mOn the Foundations of Deep Learning: SGD, Overparametrization, and GeneralizationJason Lee · Frontiers of Deep Learning 2019
• 01h 17mImplicit Regularization INati Srebro · Deep Learning Boot Camp 2019
• 50mFrom Classical Statistics to Modern Machine LearningMikhail Belkin · Frontiers of Deep Learning 2019
• 01h 01mOptimization's Untold Gift to Learning: Implicit RegularizationNathan Srebro Bartom · Optimization, Statistics and Uncertainty
• 01h 14mRecent Developments in Over-parametrized Neural Networks, Part IJason Lee · Deep Learning Boot Camp 2019
• 50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond KernelsYuanzhi Li · Frontiers of Deep Learning 2019
• 01h 23mImplicit Regularization IINati Srebro · Deep Learning Boot Camp 2019
• 44mMachine Learning ExposedJames Weaver · Codemotion Amsterdam 2017