Learning Functions and Sets with Spectral Regularization
summary
Presents a concrete alternative to penalized learning: replace ridge/Tikhonov by spectral filter functions and implement them by early-stopped iterations (gradient descent / conjugate gradient), which exactly reproduce many regularizers without computing an SVD and cut computational cost from O(n^3) SVD down to iterative costs (depends on iterations t, often ≪ n). Gives kernelized implementation (use kernel matrix, representer trick) and an iterative regularization-as-computation recipe for support estimation via a stabilized Mercer expansion; proves consistency, notes Gaussian kernels fail to separate some sets, and reports good empirical behavior.
AI-written summary · about
chapters (17)
- 0:00Start
- 1:03Signal Classification
- 3:12III-Posed Inverse Problems
- 6:33Spectral Fitering
- 19:56Supervised Learning
- 21:01Toy Case: Linear Models
- 24:45Non-Linear, Nonparametric Models
- 27:50Algorithms
- 30:45Theory
- 33:30Learning and Inverse Problems
- 34:14What's up now?
- 35:40Other Learning Problems
- 37:01Learning Sets
- 37:46Setting
- 38:12Mercer Theorem
- 41:31Spectral Characterization of the Support
- 43:11Conclusion
related talks
29mImplicit Regularization in Nonconvex Statistical Estimation
01h 05mFirst-Order Optimization and Online Learning Techniques in the Design of Fast Spectral Algorithms
33mThe Power and Limitations of Kernel Learning
01h 05mOptimization for Machine Learning I
01h 23mImplicit Regularization II
32mRegularized Nonlinear Acceleration
01h 06mAnalytic Methods for Supervised Learning III
38mWhy Deep Learning Works: Implicit Self-Regularization in Deep Neural Networks