Foundations of RL
summary
Covers three concrete algorithms: TD learning, Q‑learning, and Natural Policy Gradient, with claims of finite‑time bounds for linear function approximation (Sutton 1988; Tsitsiklis & Van Roy late 1990s) and an NTK‑style extension to wide neural nets. Shows Q‑function fixed‑point is a projection of Tπ, explains why the Bellman max breaks 2‑norm proofs and becomes a p‑norm contraction for large p, and describes practical fixes — target networks and batched “freeze‑then‑swap” updates — plus known failure cases (Baird) and open problems about depth, sample complexity and last‑iterate guarantees.
AI-written summary · about
chapters (23)
- 0:00Linear function approximation
- 2:42Extending to neural networks
- 4:05Gradient descent perspective
- 5:25TD learning approximation
- 7:36Vector form analysis
- 10:51Convergence and bootstrapping
- 12:20Fixed point equation
- 16:31Projection operator analysis
- 18:21Neural network approximation
- 23:07Convergence rates
- 24:38Depth in neural networks
- 27:32Finding optimal policies
- 30:04Q-learning and P-norms
- 36:13P-smoothness property
- 40:31Linear approximation issues
- 45:46Target networks and stability
- 50:36Policy evaluation errors
- 52:12Policy iteration methods
- 55:24Natural policy gradient
- 57:29Lyapunov function for policy
- 1:02:11Convergence to optimal policy
- 1:05:54Policy iteration robustness
- 1:07:31Open problems and future work
related talks
56mA Tutorial on Reinforcement Learning II
33mBreakthroughs and the Future of (Deep) Reinforcement Learning
11mLinear Predictors in Reinforcement Learning: ...
49mML: Teaching machine to learn from its own mistakes (...)
42mReinforcement Learning using Generative Models for Continuous State and Action Space Systems
02h 42m[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents
41mBetter Learning from the Past: Counterfactual / Batch RL
48mPractical Model-based Algorithms for Reinforcement Learning and Imitation Learning, with...