Unexpected Test Losses from Generalization Theory?
summary
Shows that overfitting can change the effective test loss: empirical L1 training behaves like a Huber and then a squared loss as capacity grows, and a ReLU interpolator converges to a discontinuous pre-activation loss rather than a squared one. Analysis uses Moreau envelope, CGMT/AMP reductions and Rademacher complexity; experiments (200 samples, up to 800 features via CVXPY) reproduce an intercept shift and benign overfitting. Bounds are tight in proportional asymptotics, predict the LAD oracle ≈0.798σ (not σ), and rely on Gaussian / multi‑index assumptions. square-root Lipschitz
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:03Generalization theory
- 6:28Unexpected phenomena
- 11:07Income prediction example
- 20:35Optimal generalization bounds
- 33:44Moreau envelopes literature
- 36:33Square root Lipschitz losses
- 47:24ReLU regression losses
- 51:00Proof techniques
- 57:08Closing thoughts
- 58:14Q&A
related talks
01h 20mThe elusive generalization: classical bounds to double descent to grokking
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
01h 23mGeneralization II
06mNeural Networks Demystified: Part 7
55mThe Devil is in the Tails and Other Stories of Interpolation
48mBenign Overfitting in Linear Prediction
01h 17mTowards Understanding Generalization Properties of Score-Based Losses
50mFrom Classical Statistics to Modern Machine Learning