Tutorial: Statistical Learning Theory and Neural Networks II
summary
Shows that gradient-based training either satisfies a Polyak–Lojasiewicz (PL) inequality (so stationary points imply small empirical exponential loss) under two concrete regimes or fails: (1) the neural tangent kernel regime uses M → large law-of-large-numbers to replace gradients by an NTK, producing a PL constant ~γ/2 when an infinite-width feature classifier attains margin γ; (2) the feature-learning regime gives a direct PL bound α·γ for two‑layer nets with gradient margin ≥γ and leaky activations φ′≥α, so optimization provably reaches low training loss.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:09Neural Network Optimization
- 0:43Refresher on Convexity
- 1:17Gradient Descent with the Fixed Learning Rate
- 10:39Gradient Margin
- 13:17Gradient of the Network at Initialization
- 19:36The Neural Tangent Kernel
- 42:45Leaky Activations
related talks
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
50mFrom Classical Statistics to Modern Machine Learning
01h 14mRecent Developments in Over-parametrized Neural Networks, Part II
01h 20mThe elusive generalization: classical bounds to double descent to grokking
50mLearning and Generalization in Over-parametrized Neural Networks, Going Beyond Kernels
13mThe dynamical view of machine learning: implicit biases of large learning rates
01h 23mImplicit Regularization II
41mTransfer learning via local convergence rates of the nonparametric least squares estimator