Generalization IV
summary
Compares K‑nearest neighbors and kernel smoothing with explicit bias–variance decompositions, deriving the minimax rate N^{-2/(2+d)} under Lipschitz assumptions, and shows kernel methods with singular kernels and simplicial interpolation can still achieve good risk. Describes empirical facts that nets and kernels interpolate to zero training loss even with noisy labels, analyzes minimum‑norm interpolants in RKHS and linear regression, and gives an if‑and‑only‑if condition on covariance eigenvalues (effective ranks) for benign overfitting.
AI-written summary · about
chapters (14)
- 0:00Generalization
- 1:10Biasvariance decomposition
- 5:45Square loss
- 12:02XS risk
- 15:12Upper bound
- 24:24Kernel smoothing
- 27:00estimator
- 29:15bias
- 31:00conditional expectation
- 36:08regression loss
- 41:11surprise
- 43:18adaboost
- 48:56kernel regression
- 52:55one nearest neighbor
related talks
33mThe Power and Limitations of Kernel Learning
50mFrom Classical Statistics to Modern Machine Learning
48mBenign Overfitting in Linear Prediction
44mConcentration Inequalites in Learning with Kernels
01h 02mPrediction, Generalization, Complexity: Revisiting the Classical View from Statistics Part 2
41mTransfer learning via local convergence rates of the nonparametric least squares estimator
01h 04mUniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting
46mSketching Techniques for Kernel Density Estimation