Regularized Nonlinear Acceleration
summary
Post-processes the last K iterates (typically 5–10) by forming increments, solving a small Tikhonov-regularized linear system, and returning a C-weighted average as the new estimate. Claims Tikhonov-regularized minimal polynomial extrapolation stabilizes Krylov ill-conditioning, and solve a 5x5 regularized linear system costs negligible extra time. Shows on quadratics the procedure recovers Nesterov accelerated rate on quadratics and examples (Aitken, epsilon algorithm) gain several digits from three iterates. Requires tuning the regularization parameter—requires lambda calibration via grid search—and degrades when stochastic noise dominates.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 6:55Minimal Polynomial extrapolation
- 11:53Convergence rate
- 22:00Results
- 26:26Online Stochastic Gradient
- 27:11Network Acceleration
- 28:10Conclusion
related talks
01h 18mOptimization I
01h 23mImplicit Regularization II
46mLearning Functions and Sets with Spectral Regularization
01h 07mOptimization Crash Course (continued)
54mHalpern Iteration and Equilibria Problems
01h 06mAnalytic Methods for Supervised Learning III
01h 07mStochastic Second Order Optimization Methods I
01h 05mFirst-Order Optimization and Online Learning Techniques in the Design of Fast Spectral Algorithms