Data Efficient Methods for Reinforcement Learning

on YouTube ↗11m

" summary

Claims provable speedups for SGD with experience replay on an autoregressive Markov chain showing loss decays exponentially with an improved dependence of T over the square root of the mixing time versus sequential SGD’s dependence over the mixing time, a diagnosis that Fitted Q‑Evaluation can diverge under common MLP architectures and fixes that stabilize it (ResNet, distributional FQE, shared representations, hypermodels) without losing accuracy, and two generative‑model algorithms that compute ε‑optimal policies with near‑linear runtime and logarithmic-in-1/ε sample complexity and a second near‑optimal cubic dependence on 1/(1−γ).

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 0:31Challenges
3. 0:50Problem Setting
4. 1:59Experience Replay
5. 3:09Results
6. 3:51Distribution Shift
7. 5:12Algorithms
8. 5:43Better architectures
9. 6:48General data efficient algorithms
10. 8:04Conclusion

" related talks

• 01h 05mFoundations of RLRayadurgam Srikant · Data-Driven Decision Processes Boot Camp 2022
• 42mReinforcement Learning using Generative Models for Continuous State and Action Space SystemsRahul Jain · Reinforcement Learning from Batch Data and Simulation
• 56mA Tutorial on Reinforcement Learning IIEmma Brunskill · Foundations of Machine Learning Boot Camp 2017
• 28mOptimizing Long-Tail and MoE Challenges in Reinforcement Learning with SGLangChenyang Zhao · PyTorch Conference 2025
• 01h 01mRobustness, Stochastics, Uncertainty 2Sasha Rakhlin, Ben Recht, Laurent El Ghaoui · Bridging Continuous and Discrete Optimization Boot Camp 2017
• 01h 17mOnline Reinforcement Learning and RegretSean Sinclair, Christina Lee Yu · Data-Driven Decision Processes Boot Camp 2022
• 33mTradeoffs in Large Scale Learning: Statistical Accuracy vs. Numerical PrecisionSham Kakade · Information Theory, Learning and Big Data (Simons Institute, 2015)
• 55mOptimality and Approximation with Policy Gradient Methods in Markov Decision ProcessesAlekh Agarwal · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)