Better Learning from the Past: Counterfactual / Batch RL

on YouTube ↗41m

" summary

Counterfactual batch RL techniques estimate and optimize policies from logged data using importance-weighted stationary-distribution ratios and doubly robust estimators to correct for state–action mismatch, targeting global selection of the best policy in a class with regret-style rates. Experiments show a policy gradient variant learns near-optimal control from poor behavior data in cart-pole, and an 11,000 trajectories offline model predicts a 30% predicted / 32% observed boost in student persistence when deployed, using interpretable linear threshold policies.

AI-written summary · about

" chapters (19)

1. 0:00Intro
2. 0:53Sequential Decision Making Under Uncertainty
3. 2:01Learning to Make Good Sequences of Decisions Under Uncertainty → 1980s Reinforcement Learning
4. 5:20Background: Markov Decision Process Value Function
5. 5:32Background: Reinforcement Learning
6. 7:55Counterfactual / Batch Off Policy Reinforcement Learning
7. 9:34Need for Generalization
8. 10:19Growing Interest in Causal Inference & ML
9. 10:47Batch / Counterfactual Policy Optimization: Pick Policy w/Best Estimated Expected Sum of Rewards
10. 12:49Quest: Batch Policy Optimization w/ Generalization Bounds
11. 15:28Challenge: Good Error Bound Analysis
12. 18:19Aim: Strong Generalization Guarantees on Policy Performance, Alternative: Guarantee Find Good in Class Policy
13. 18:49Off-Policy Policy Gradient with State Distribution Correction
14. 22:22Aim: Strong Generalization Guarantees on Policy Performance, Alternative: Guarantee Find Best in Class Policy
15. 23:32Example: Linear Thresholding Policies Starting HIV treatment as soon as
16. 25:46Use an Advantage Decomposition
17. 26:31Use a Doubly Robust Advantage Decomposition
18. 31:23Quest for Batch Policy Optimization with Generalization Guarantees
19. 35:48Techniques to Minimize & Understand Data Needed to Learn to Make Good Decisions

" related talks

• 35mOnline Learning with A Lot of Batch DataShie Mannor · Reinforcement Learning from Batch Data and Simulation
• 56mA Tutorial on Reinforcement Learning IIEmma Brunskill · Foundations of Machine Learning Boot Camp 2017
• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 11mLinear Predictors in Reinforcement Learning: ...Andrea Zanette · Meet the Fellows Welcome Event
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 01h 05mFoundations of RLRayadurgam Srikant · Data-Driven Decision Processes Boot Camp 2022
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 31mOnline Reinforcement Learning With The Help Of Confounded Offline DataUri Shalit · Learning from Interventions 2022