Offline Reinforcement Learning and Model-Based Optimization

on YouTube ↗34m

" summary

Identifies the core failure mode as distributional shift from optimizing learned models and shows two concrete fixes with results: a regularized Q‑learning objective called conservative Q‑learning that adversarially finds and pushes down spuriously large Q values and provably yields a lower bound in expectation, improving scores on D4RL benchmarks (notably maze stitching, dexterous manipulation, Atari), and a model inversion networks approach that trains a conditional GAN X = G(Y,Z) and selects Y,Z so forward and inverse models agree, demonstrated on protein design and face synthesis.

AI-written summary · about

" chapters (23)

1. 0:00Intro
2. 2:00What makes modern machine learning word
3. 2:33Predictive models are very powerful!
4. 4:08Automated decision making is very powerf
5. 5:06First setting: data-driven reinforcement lear
6. 6:14Second setting: data-driven model-based optimization
7. 10:40Off-policy RL: a quick primer
8. 11:29What's the problem?
9. 12:51Distribution shift in a nutshell
10. 15:55How do prior methods address this?
11. 19:26Learning with Q-function lower bounds Algorithm
12. 21:30Does the bound hold in practice?
13. 22:37How does CQL compare?
14. 24:04Predictive modeling and design
15. 24:30What's wrong with just doing prediction?
16. 24:44The model-based optimization problem
17. 25:38Uncertainty and extrapolation
18. 26:40What can we do?
19. 27:56Model inversion networks (MINS)
20. 30:09Putting it all together
21. 30:23Experimental results
22. 30:59Some takeaways
23. 31:38Some concluding remarks

" related talks

• 48mPractical Model-based Algorithms for Reinforcement Learning and Imitation Learning, with...Tengyu Ma · Frontiers of Deep Learning 2019
• 56mA Tutorial on Reinforcement Learning IIEmma Brunskill · Foundations of Machine Learning Boot Camp 2017
• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 09mLearning Under RequirementsLuiz Chamon · Meet the Fellows Welcome Event
• 30mOffline RL with RLlibEdi Palencia · Ray Summit 2021
• 11mLinear Predictors in Reinforcement Learning: ...Andrea Zanette · Meet the Fellows Welcome Event
• 48mOffline Reinforcement Learning With Only RealizabilityJason Lee · Adversarial Approaches in Machine Learning 2022