Optimization for Machine Learning II
summary
AdaGrad is presented as the adaptive regularizer used in practice for deep nets; stochastic gradient descent needs O(1/epsilon^2) iterations while smooth full gradient descent needs O(1/epsilon). A hybrid variance reduction method (SVRG) uses occasional full-gradient passes to get log(1/epsilon) passes plus O(sqrt(M/epsilon)) work, beating pure SGD. A linear-time stochastic Newton estimator samples rank-one Hessians via a randomized series; constrained problems use Frank-Wolfe to avoid costly projections.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:56Accelerating gradient descent?
- 5:14Condition number of convex functions
- 8:37Examples
- 9:25Smooth gradient descent
- 16:48Non-convex stochastic gradient descent
- 26:39Controlling the variance: Interpolating GD and SGD
- 33:16Acceleration/momentum (Nesterov '83)
- 35:10Experiments w. convex losses
- 38:34Higher Order Optimization
- 42:25Stochastic Newton?
- 44:55Circumvent Hessian creation and inversion!
- 55:32Recommendation systems
- 56:54Bounded trace norm matrices
- 58:05Conditional Gradient algorithm Frank, Wolfe '56 Convex opt problem
related talks
58mFirst-Order Stochastic Optimization
41mOptimization Algorithms for Deep Learning
01h 07mOptimization Crash Course (continued)
01h 01mRobustness, Stochastics, Uncertainty 2
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
01h 18mOptimization I
33mStochastic Optimization for Big Data Machine Learning Problems
29mImplicit Regularization in Nonconvex Statistical Estimation