Fundamentals of Markov Decision Processes
summary
Covers finite-state, finite-action discounted MDPs with gamma in (0,1); derives the Bellman equation and shows the Bellman operator is a contraction so the fixed point is unique and computable by iteration. Gives concrete algorithms: value iteration converges at geometric rate gamma with an explicit supremum-norm error bound, and policy iteration halts in a finite number of policy upgrades with monotonic improvement. Introduces Q-functions as action-value targets usable when transition probabilities are unknown.
AI-written summary · about
chapters (18)
- 0:00Start
- 0:15Fundamentals of Markov Decision Processes
- 0:37Basics of Markov Decision Processes
- 0:52What Is the Mdp
- 6:27Important Concepts in the Markov Decision Process
- 6:56Reward Function
- 7:30General Notation for a Markov Decision Process
- 7:40Infinite Time Horizon
- 9:43Stationary Policies
- 11:46Objective Function
- 24:08Rewrite the Bellman Equation
- 27:31Contraction Mapping
- 35:43Policy Iteration Algorithm
- 36:14Value Evaluation
- 38:56Policy Improvement
- 39:12Instantaneous Reward
- 47:16The True Function
- 49:55The Optimal Q Function
related talks
55mOptimality and Approximation with Policy Gradient Methods in Markov Decision Processes
01h 01mA Tutorial on Reinforcement Learning I
52mThe Simplex and Policy-Iteration Methods are Strongly Polynomial...
04m1-2-3-Go! Policy Synthesis for Parameterized Markov Decision Processes via(…)
20mThe Complexity of Infinite-Horizon General-Sum Stochastic Games
33mFormal Verification of Probabilistic Deep Reinforcement Learning Policies with(…)
31mLearning to Control Unknown Systems
46mMarkov Persuasion Process and its Reinforcement Learning