Optimization Algorithms for Deep Learning
summary
Covers why plain gradient descent stalls on flat or tiny-gradient regions and how practical fixes speed training: momentum + Nesterov to push through plateaus, AdaGrad → RMSProp to adapt and decay per-parameter rates, and Adam (Kingma & Ba, 2015) which combines momentum and RMSProp with bias correction. Demonstrations use R/TensorFlow code and standard hyperparameters beta1, beta2, epsilon: unnormalized GD needs ~100 million epochs versus Adam ~100,000; with normalization GD ~2,800 epochs and Adam ~500.
AI-written summary · about
chapters (11)
- 0:00Optimization Algorithms for Deep Learning
- 7:47Gradient Descent Algorithm?
- 14:45Solutions to Gradient Descent Algorithm Problem
- 15:46First Solution to the Gradient Descent Algorithm Problem
- 17:24Exponentially Weighted Moving Average of Gradients
- 21:49Solution to Momentum Nesterov Momentum
- 24:10Adaptive Gradient Algorithm
- 26:32Adaptive Moment Estimation
- 31:54Adam Implementation
- 32:14FINAL RESULTS
- 37:17Summary
related talks
46mAnalyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
01h 04mOptimization for Machine Learning II
46mGradient Descent: The Ultimate Optimizer
01h 18mOptimization I
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
46mOn the Foundations of Deep Learning: SGD, Overparametrization, and Generalization
16mSDE Approximations and Scaling Rules for Distributed Deep Learning
01h 04mPredicting and optimizing the behavior of large ML models