Reinforcement Learning from Batch Data and Simulation
2020-11-30..2020-12-04
-- online --
[Multi-Agent Reinforcement Learning and Bandit Learning, Mathematics of Online Decision Making 2020, Quantifying Uncertainty: Stochastic, Adversarial, and Beyond, Adversarial Approaches in Machine Learning 2022, Emerging Challenges in Deep Learning (Simons Institute workshop, 2019), Learning from Heterogeneous Sources 2026]
who it's for
reinforcement learning researchers, offline reinforcement learning practitioners, reinforcement learning engineers, control systems engineers, simulation and robotics engineers, statistical learning theorists
topics
offline policy evaluation in tabular RL; self-normalized importance weighting; prioritized experience replay as modified loss; batch value-function tournament methods; generative-model RL for continuous control; zap stochastic approximation for Q-learning; relative Q-learning for bounded variance; multiagent Q decomposition (CollabQ); Monte Carlo discretization for multistage programs; model-based ERM for offline learning; learning switched linear dynamical systems via Hankel matrix
| title | author | length |
|---|---|---|
| Online Learning with A Lot of Batch Data | Shie Mannor | 35m |
Claims that large offline batches shrink online complexity: with only L of d context features observed in logs, least-squares + an optimism-based algorithm… | ||
| Learning Multi-Agent Collaborations With Decomposition | Yuandong Tian | 30m |
Three papers present concrete algorithms, proofs and results: CollabQ decomposes joint Q into a local Q-alone and a co-collaboration term learned from local… | ||
| Beyond Worst-Case: Instance-Dependent Optimality in Reinforcement Learning | Martin Wainwright | 44m |
Derives asymptotic and non‑asymptotic instance‑dependent lower bounds for tabular policy evaluation, identifying an optimal covariance matrix that determines… | ||
| An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay | Doina Precup | 26m |
Claims that prioritized experience replay is equivalent in expectation to uniform sampling with a modified loss, derives the equivalence via importance… | ||
| Confident Off-policy Evaluation and Selection through Self-Normalized Importance Weighting | Ilja Kuzborskij | 41m |
Derives non-asymptotic, high-probability, data-dependent lower bounds for self-normalized importance weighting in contextual bandits, using a semi-empirical… | ||
| Robust Learning of Stochastic Dynamical Systems | Munther Dahleh | 33m |
Claims a practical method to learn switched stochastic linear systems when mode labels are observed, producing a low-order approximation selected from the data… | ||
| Monte Carlo Sampling Approach to Solving Stochastic Multistage Programs | Alex Shapiro | 32m |
Argues that multistage stochastic linear programs are tractable with Monte Carlo discretization when decisions do not affect the data distribution, using… | ||
| Reinforcement Learning using Generative Models for Continuous State and Action Space Systems | Rahul Jain | 42m |
Algorithm RandPaul uses a generative simulator, empirical Bellman updates and an actor‑critic with randomized shallow nets (only last layer learned) and sample… | ||
| Batch Value-function Approximation with Only Realizability | Nan Jiang | 31m |
Claims a polynomial sample complexity guarantee under only realizability by introducing a batch value-function tournament that compares every pair f, f' and… | ||
| Uniform Offline Policy Evaluation (OPE) and Offline Learning in Tabular RL | Yu-Xiang Wang | 34m |
Analyzes offline RL in episodic tabular (time-inhomogeneous) MDPs with N logged trajectories under a coverage assumption and proves pointwise OPE achieves the… | ||
| Zap Stochastic Approximation and Implications to Q-Learning | Sean Meyn | 29m |
Introduces Zap stochastic approximation as a two-time-scale root-finding scheme that uses an online matrix gain (approximate inverse Jacobian) to force… | ||
| Q-learning with Uniformly Bounded Variance | Adithya Devraj | 31m |
Proposes a simple modification that subtracts a learned constant from the Bellman update so the linearization gets a rank‑one shift, moving the problematic… | ||
| The Mean-Squared Error of Double Q-Learning | R. Srikant | 35m |
Claim: averaging the two maintained Q estimates and using twice the step size (2G/(n+1)) makes Double Q-learning match the asymptotic mean-squared error of… | ||
| Offline Reinforcement Learning and Model-Based Optimization | Sergey Levine | 34m |
Identifies the core failure mode as distributional shift from optimizing learned models and shows two concrete fixes with results: a regularized Q‑learning… | ||
| Convergence and Sample Complexity of Gradient Methods for the Model-Free Linear Quadratic Regulator | Mihailo Jovanovic | 33m |
Proves exponential stability of gradient flow over the set of stabilizing gains and shows gradient descent with fixed step attains linear convergence; uses a… | ||
| Policy Evaluation under Interference | Stefan Wager | 30m |
Compares two concrete interference approaches: a graphon central limit theorem for network interference that gives root-n convergence for the direct effect… | ||
| Stable Reinforcement Learning with Unbounded State Space | Qiaomin Xie | 31m |
Algorithm uses a Monte Carlo oracle that takes a single queried state, runs sparse sampling or MCTS rollouts to estimate multi-step Q-values, and returns a… | ||
| Panel Discussion | Sean Meyn, Emma Brunskill | 45m |
Panelists argue that in the next five years batch/offline RL will move from demos to real-world proposals for policy design (e.g. tax or macro suggestions) and… | ||
| Multiagent Reinforcement Learning: Rollout and Policy Iteration | Dimitri Bertsekas | 37m |
A new multiagent policy-iteration method tests one agent at a time instead of all joint actions, using dynamic programming plus simulation and neural value and… | ||
| nothing matches | ||