From Classical Statistics to Modern Machine Learning
summary
Argues that modern ML succeeds because many methods interpolate the training set and still generalize: kernels and nets fit MNIST with 60% random labels yet a Laplace-kernel fit reaches ~1e-27 L2 loss and test accuracy stays far above random. Describes a double descent risk curve that falls again after the interpolation threshold as width increases, with random-Fourier/ReLU features converging to a minimum-norm kernel limit. Shows interpolation makes SGD fast (small-batch exponential convergence) and can create dense adversarial points when labels are noisy.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:07Supervised ML
- 2:43Generalization bounds
- 3:34Classical U-shaped generalization curve
- 4:21Does interpolation overfit?
- 4:59Interpolation does not overfit even for very noisy data
- 6:52Deep learning practice
- 10:46Generalization theory for interpolation?
- 12:24A way forward?
- 14:19Interpolated k-NN schemes
- 16:13Interpolation and adversarial examples
- 18:41"Double descent" risk curve
- 24:07what is the mechanism?
- 28:48Double Descent in Linear regression
- 29:07Occams's razor
- 30:29The landscape of generalization
- 35:20where is the interpolation threshold?
- 36:42Optimization under interpolation
- 37:31SGD under interpolation
- 39:09The power of interpolation
- 41:40Learning from deep learning: fast and effective kernel machines
- 42:21Important points
- 43:40From classical statistics to modern ML
related talks
01h 22mGeneralization IV
26mIs Overfitting Actually Benign? On the Consistency of Interpolating Methods
01h 02mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2
01h 03mOverview of Statistical Learning Theory Part 2
01h 02mTutorial: Statistical Learning Theory and Neural Networks II
01h 20mThe elusive generalization: classical bounds to double descent to grokking
01h 37mTutorial: Implicit Bias II
55mThe Devil is in the Tails and Other Stories of Interpolation