Gradient Descent: The Ultimate Optimizer
summary
Claims that hyper-parameters and even hyper‑hyper‑parameters are learned by gradient descent, demonstrated with concrete Kotlin code that computes derivatives via dual numbers (extended to vectors of epsilons for many variables) and then optimized the step sizes. A small example learns G ≈ 9.8 and step sizes converge quickly; stacking four optimizer levels makes performance insensitive to initial η. Implementation notes: initial method is quadratic, a continuation-based AD variant makes it linear time and practical.
AI-written summary · about
chapters (4)
- 0:00Start
- 7:21The Magic of Duality
- 20:52Most Trivial Example
- 32:08High school math review: Product Rule
related talks
29mGradient descent is taking away your Android developer job
41mOptimization Algorithms for Deep Learning
01h 18mOptimization I
16mAlgorithmic Differentiation: C++ & Extremum Estimation
01h 07mOptimization Crash Course (continued)
01h 01mOptimization's Untold Gift to Learning: Implicit Regularization
01h 04mOptimization for Machine Learning II
05mNon-monotonic convergence of gradient descent with large stepsize