Zap Stochastic Approximation and Implications to Q-Learning
summary
Introduces Zap stochastic approximation as a two-time-scale root-finding scheme that uses an online matrix gain (approximate inverse Jacobian) to force near-linear ODE dynamics, proving near-universal stability and the optimal covariance CLT under assumptions. Theory requires alpha=1/n and a faster tracking step gamma_n ~ 1/n^rho (rho in (1/2,1)) so gamma_n/alpha_n → ∞; implementation needs access to derivatives and regularization when the Jacobian is singular. Empirically it runs on neural-network Q-learning (mountain car, cartpole, acrobot) with reliable performance and little tuning.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 2:33Stochastic Approximation
- 17:26Zap Stochastic Approximation
- 19:55Matrix Gain
- 25:37Example
- 27:00Conclusion
related talks
01h 05mFoundations of RL
15mMulti-Agent Reinforcement Learning In Stochastic Games: From Alphago To Robust Control
11mData Efficient Methods for Reinforcement Learning
16mSDE Approximations and Scaling Rules for Distributed Deep Learning
01h 18mOptimization I
01h 01mRobustness, Stochastics, Uncertainty 2
01h 07mOptimization Crash Course (continued)
01h 07mStochastic Second Order Optimization Methods I