Why Deep Learning Works: Implicit Self-Regularization in Deep Neural Networks
summary
Claim: deep nets develop implicit self-regularization visible in the singular-value spectra of layer weight matrices. Empirical toolset uses heavy-tailed random matrix theory to classify spectra into five-plus-one phases (random-like → bulk+spikes → heavy-tailed → rank collapse). Specific findings: smaller SGD batch sizes push models toward heavy-tailed spectra, reduce stable rank, and improve test accuracy; large batches keep random-like spectra and worsen generalization. The analysis rests on eigen/singular-value bulk and edge statistics.
AI-written summary · about
chapters (9)
- 0:00Introduction to the topic
- 1:46A practical theory
- 6:31Understanding regularization
- 8:37Spectral analysis techniques
- 12:35Random matrix theory basics
- 17:45Model analysis examples
- 20:20Phases of learning theory
- 25:37Batch size experimentation
- 31:23Conclusion and Q&A
related talks
11mOn Implicit Regularization in Deep Learning
23mOn Implicit Regularization in Deep Learning
01h 17mImplicit Regularization I
29mImplicit Regularization in Nonconvex Statistical Estimation
01h 02mTutorial on Deep Learning I
01h 21mTutorial: Implicit Bias I
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
13mThe dynamical view of machine learning: implicit biases of large learning rates