Offline Reinforcement Learning With Only Realizability

on YouTube ↗48m

" summary

Proposes solving offline RL by solving a regularized primal-dual LP (density-regularized DICE-style objective) to recover an importance weight w and extract a policy, requiring only realizability for value and density ratio plus a single-policy concentrability on the regularized occupancy. Shows unbiased sample estimates, uniform-convergence bounds giving sample rates that scale with log |function classes| and polynomially in 1/ε (a provable ε^-6 bound with an ε^-4 asymptotic improvement), and handles agnostic misspecification and policy extraction via behavior cloning.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 5:00Linear Kernel
3. 9:00Offline vs Online
4. 13:40What should F approximate
5. 15:36What is completeness
6. 17:48Common justification for completeness
7. 19:24Classical pessimism
8. 25:02Linear pessimism
9. 29:28Density regularization
10. 32:18Policy extraction
11. 33:24Realizability
12. 40:30Proof
13. 41:40Other Results
14. 43:45Conclusion

" related talks

• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 55mWhat Are the Statistical Limits of Offline Reinforcement Learning With Function Approximation?Sham Kakade · Mathematics of Online Decision Making 2020
• 34mUniform Offline Policy Evaluation (OPE) and Offline Learning in Tabular RLYu-Xiang Wang · Reinforcement Learning from Batch Data and Simulation
• 11mLinear Predictors in Reinforcement Learning: ...Andrea Zanette · Meet the Fellows Welcome Event
• 30mOffline RL with RLlibEdi Palencia · Ray Summit 2021
• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 54mOn the Hardness of Reinforcement Learning With Value-function ApproximationNan Jiang · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 35mOnline Learning with A Lot of Batch DataShie Mannor · Reinforcement Learning from Batch Data and Simulation