Model-Based Reinforcement Learning with Value-Targeted Regression

on YouTube ↗36m

" summary

Proposes using value-targeted regression: fit transition models only to predict next-step value estimates, then build confidence sets over models and do optimistic planning with those models. Claims this avoids state reconstruction (motivated by MuZero) and gives provable online regret bounds that scale with sqrt(T) and with model complexity measured by the eluder dimension and log covering numbers of the transition class. Special cases recover linear-model rates (dimension dependence, reducible to sparsity) and match known Bayesian bounds.

AI-written summary · about

" chapters (16)

1. 0:00Intro
2. 1:22Model-Based Reinforcement Learning
3. 4:01Episodic Reinforcement Learning
4. 5:13Upper Confidence Model-Based RL (UCRL)
5. 7:08The class of deterministic continuous systems . Consider a deterministic system
6. 8:03A Simple Metric-Based RL Algorithm
7. 11:07Doubling Dimension d
8. 12:19Feature space embedding of transition model
9. 13:41The MatrixRL Algorithm
10. 15:11From Feature to Kernel Embedding of Transition Model
11. 17:31A motivating example: MuZero
12. 21:20Assumption of Value-Targeted Regression
13. 22:20Value-Targeted Regression (VTR) for Confidence Set Construction
14. 25:31Full Algorithm of UCRL-VTR
15. 26:30Regret analysis of UCRL-VTR
16. 28:31A Special Case

" related talks

• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 56mA Tutorial on Reinforcement Learning IIEmma Brunskill · Foundations of Machine Learning Boot Camp 2017
• 48mPractical Model-based Algorithms for Reinforcement Learning and Imitation Learning, with...Tengyu Ma · Frontiers of Deep Learning 2019
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 41mBetter Learning from the Past: Counterfactual / Batch RLEmma Brunskill · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 53mOff-policy Policy OptimizationDale Schuurmans · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 35mJe ne regrette rien: Teaching Machine Learning Models Regret AvoidanceLaura Israel · PyData Amsterdam 2024