Offline RL with RLlib

on YouTube ↗30m

" summary

RLlib is extended to run Conservative Q‑Learning (CQL) by modifying DQN losses and reusing existing RLlib components and CLI. A dataset generator produces datasets with five levels of optimality. Demo shows an offline brain matched online performance with 10x less data, while online training needed 100,000 samples to converge. Bad data still breaks policies. Evaluation uses FQE or model‑based off‑policy evaluation and distribution shift plus lack of exploration remain practical challenges.

AI-written summary · about

" chapters (16)

1. 0:00Intro
2. 1:00About Project Bonsai
3. 1:42Journey to create autonomous systems
4. 3:08Machine teaching
5. 4:15Machine learning
6. 6:14Open source
7. 7:17Requirements
8. 11:09Challenges
9. 13:13What is Offline
10. 13:33Offline vs Online RL
11. 16:54Offline vs Machine Learning
12. 18:40Leveraging Early
13. 19:19CQL
14. 22:29SQL
15. 24:47Demo
16. 27:22Whats next

" related talks

• 34mOffline Reinforcement Learning and Model-Based OptimizationSergey Levine · Reinforcement Learning from Batch Data and Simulation
• 48mOffline Reinforcement Learning With Only RealizabilityJason Lee · Adversarial Approaches in Machine Learning 2022
• 54mGeneralization and Robustness in Offline Reinforcement LearningWen Sun · Quantifying Uncertainty: Stochastic, Adversarial, and Beyond
• 41mBetter Learning from the Past: Counterfactual / Batch RLEmma Brunskill · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 01h 11mFoundations of RLRayadurgam Srikant · Data-Driven Decision Processes Boot Camp 2022
• 31mOnline Reinforcement Learning With The Help Of Confounded Offline DataUri Shalit · Learning from Interventions 2022
• 11mLinear Predictors in Reinforcement Learning: ...Andrea Zanette · Meet the Fellows Welcome Event
• 33mBreakthroughs and the Future of (Deep) Reinforcement LearningAndreas Buhlmeier · Big Data Conference Europe 2019