Foundations of RL
summary
Covers concrete algorithms and finite-time claims: presents TD learning, Q‑Learning and Natural Policy Gradient and shows how stochastic‑approximation tools prove convergence in the tabular case. Builds the asynchronous update rule, treats Markovian noise by conditioning on a tau (mixing) window (tau ≈ O(log(1/ε))), and derives that fixed stepsize yields O(ε) error while 1/K stepsizes drive error to zero; requires an exploratory randomized policy to visit every state–action pair. Sketches extension to function approximation and cites recent (≈2019) finite‑time analyses.
AI-written summary · about
chapters (13)
- 0:00Start
- 0:35Reinforcement Learning Problem
- 2:47Function Approximation
- 3:09The Algorithm
- 5:55Rationale behind this Algorithm
- 7:49Stochastic Approximation
- 11:42Vector Notation
- 17:57Temporal Difference Algorithm
- 20:01Estimate the Probability Transition Matrix
- 24:07Noise Free Analysis
- 27:11The Noise Free Model
- 27:25Potential Function
- 42:52Noisy Case
related talks
56mA Tutorial on Reinforcement Learning II
11mData Efficient Methods for Reinforcement Learning
31mLearning to Control Unknown Systems
15mMulti-Agent Reinforcement Learning In Stochastic Games: From Alphago To Robust Control
01h 01mA Tutorial on Reinforcement Learning I
41mBetter Learning from the Past: Counterfactual / Batch RL
33mBreakthroughs and the Future of (Deep) Reinforcement Learning
38mReinforcement Learning - What it is and How it Works