Stochastic Second Order Optimization Methods II
summary
Survey of practical stochastic second‑order methods claims that limited‑memory BFGS and Hessian‑vector products make curvature usable at scale, that subsampled Newton variants recover condition‑number independence in wall‑clock time, and that SVRG‑style variance reduction lets quasi‑Newton keep fixed step sizes. Key specifics: curvature updates use S/Y histories or Hessian‑times‑vector approximations, Wolfe/curvature checks enforce positive definiteness, trust‑region/cubic rules achieve provably optimal epsilon‑second‑order rates, and some implementations outperform TensorFlow baselines on million‑dimensional nets.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:22Second Order Conditions
- 2:59Approximate Optimality
- 5:18BFGS
- 6:42Newton
- 8:04Secant
- 9:06QuasiNewton
- 10:56Taylor Expansion
- 12:00Secant Condition
- 14:57David
- 16:53Trivia
- 17:37The BFGS
- 24:27Limited Memory
- 31:36Variance Reduction
- 35:24Subsamples
- 36:48Gauss Newton
- 37:11Gaussian Newton
- 40:40Natural Gradient
- 53:50Local Search
related talks
33mStochastic Optimization for Big Data Machine Learning Problems
01h 07mOptimization Crash Course (continued)
01h 01mRobustness, Stochastics, Uncertainty 2
01h 05mFirst-Order Optimization and Online Learning Techniques in the Design of Fast Spectral Algorithms
01h 18mOptimization I
42mOptimization Crash Course
01h 04mOptimization for Machine Learning II
36mA Generalization Bound for Online Variational Inference