Benign Overfitting in Linear Prediction
summary
Shows that linear regression can benignly overfit: the minimum-norm interpolator (pseudoinverse solution) attains vanishing excess risk when the covariance eigenvalues have a long flat tail and two effective-rank conditions hold (K small relative to n and n divided by an effective rank large). Proof uses random-matrix concentration under joint Gaussian/sub-Gaussian X,Y and decomposes signal vs noise; matching upper and lower bounds depend on noise scale and factors like K/n and n/RK. Consequence: needs over-parameterization with many weak directions and yields huge adversarial sensitivity.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:01Overfitting in Deep Networks
- 10:08Interpolating Prediction Rules
- 14:21Definitions
- 17:40Interpolating Linear Regression
- 20:18Notions of Effective Rank
- 22:56Benign Overfitting: A Characterization
- 28:18Benign Overfitting: Proof Ideas
- 33:09What kinds of eigenvalues?
- 37:47Implications for deep learning
- 39:27Implications for adversarial examples
- 43:30Benign Overfitting in Linear Regression
related talks
01h 04mUniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting
01h 18mReconsidering Overfitting in the Age of Overparameterized Models
26mIs Overfitting Actually Benign? On the Consistency of Interpolating Methods
01h 22mGeneralization IV
01h 02mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2
58mBenign, Tempered, or Catastrophic: A Taxonomy of Overfitting
42mToward Understanding the Extrapolation of Nonlinear Models to Unseen Domains or the Whole Domain
01h 18mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 1