An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
summary
Claims that prioritized experience replay is equivalent in expectation to uniform sampling with a modified loss, derives the equivalence via importance sampling and shows MSE with priority-by-absolute-TD maps to an L1/Huber-style loss. Implements two algorithms (a debiased PER variant and a uniform-buffer + modified-loss version called PAL) and evaluates with TD3 on MuJoCo and Double DQN on Atari. Reports consistent gains on hard MuJoCo (humanoid) and Atari (Frostbite), occasional regressions, and that PER bias points toward the median.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:59Experience Replay
- 3:25Controlling the replay buffer
- 6:04Overview
- 7:22Main idea
- 10:34An interesting situation: MSE and L, loss
- 12:20A general result
- 14:13What about the variance?
- 16:09Mujoco results
- 17:36Atari results
- 18:27Summary comparison of final performance on Atari
related talks
11mData Efficient Methods for Reinforcement Learning
53mOff-policy Policy Optimization
01h 02mSampling for Linear Algebra, Statistics, and Optimization II
01h 23mGeneralization II
56mA Tutorial on Reinforcement Learning II
29mPure Exploration Problems
01h 17mTowards Understanding Generalization Properties of Score-Based Losses
43mPrivacy Amplification by Sampling in Practice