Is Overfitting Actually Benign? On the Consistency of Interpolating Methods
summary
Claims that modern overparameterized interpolating methods fail consistency: training MLPs to interpolation on CIFAR with controlled label corruption and on synthetic 100‑dim Gaussian data (up to ~20k samples) shows test error stays away from Bayes optimal and the output bias tracks input noise. Experiments vary depth, width, loss (MSE, cross‑entropy) and initialization; interpolating networks are inconsistent in these settings and variance does not go to zero despite evidence of asymptotic unbiasedness. Kernel (RBF) bandwidth regimes remain an open test.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 0:44Teaser
- 6:40Modern Learning Methods
- 14:45Classification
- 19:12Open Questions
- 21:55Conclusion
related talks
48mBenign Overfitting in Linear Prediction
58mBenign, Tempered, or Catastrophic: A Taxonomy of Overfitting
50mFrom Classical Statistics to Modern Machine Learning
01h 18mReconsidering Overfitting in the Age of Overparameterized Models
55mThe Devil is in the Tails and Other Stories of Interpolation
01h 22mGeneralization IV
01h 02mUnexpected Test Losses from Generalization Theory?
01h 37mTutorial: Implicit Bias II