A Tutorial on Reinforcement Learning II
summary
Summarizes regret bounds and practical fixes: UCRL-2 gives regret scaling O(D · S · sqrt(A T) · log(T/δ)) under finite diameter while gap-dependent bounds can be logarithmic, and posterior/Thompson-style sampling often outperforms UCRL-2. An off‑policy case study shows a learned policy yields 30% higher engagement versus the previous best (and beats random); MAGIC blends importance sampling and models to cut mean squared error by orders of magnitude. Notes deep RL: DQN on four frames with very large nets and massive data achieves human‑level Atari and AlphaGo combines policy/value nets with MCTS.
AI-written summary · about
chapters (14)
- 0:00Intro
- 4:23UCRL2 Regret Bounds
- 9:59Bayesian Regret for RL
- 13:54Contextual Bandits: Reduction to Supervised Learning
- 19:49Off Policy Evaluation: Assume Domain Specification is correct
- 22:46MAGIC: Blend IS-Based & Model Based Estimator to Directly Min MSE
- 25:31Other Variants of Importance Sampling
- 32:12Long Horizon, Reward at End?
- 33:53Safety and Risk Sensitivity
- 36:44Scaling Up to Large Domains: Deep RL
- 38:55Brief History of Representation and RL
- 44:15Generalization & Exploration
- 46:08Representing Optimal Policy Vs Learning Optimal Policy
- 47:09Sample Efficiency & Representation
related talks
01h 17mOnline Reinforcement Learning and Regret
28mDeep Reinforcement Learning: theory, intuition, code
01h 05mFoundations of RL
34mOffline Reinforcement Learning and Model-Based Optimization
01h 11mFoundations of RL
31mOn the Complexity of Learning Good Policies With and Without Rewards
46mDynamic Regret Minimization for Bandits without Prior Knowledge
31mLearning to Control Unknown Systems