An Alternative Softmax Operator for Reinforcement Learning

on YouTube ↗31m

" summary

Introduces the Mellowmax (mellomax) operator: log of the average of e^{omega·Q}/omega (a softened Boltzmann-style average) and proves it satisfies a non-expansion contraction property so generalized value iteration has a unique fixed point and fast convergence. Experiments show Boltzmann sometimes fails (8 of 200 random MDPs nonterminating, 3 with multiple fixed points) while mellanax never does. Replacing max with mellomax in Deep Q-learning replaces max in DQN removes the target network and yields more stable learning and higher returns on several Atari-style games.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 5:05Pacman in a Grid
3. 7:47NonExpansion Property
4. 14:39Contraction Property
5. 15:15Properties
6. 16:13Algorithm
7. 17:26Bellman Operator
8. 18:54Boltzmann Operator
9. 23:01Deep Q Learning
10. 27:30Questions

" related talks

• 57mComputationally Efficient Alternatives to Nonconvex-Nonconcave Min-Max OptimizationNisheeth Vishnoi · Adversarial Approaches in Machine Learning 2022
• 55mLearning Minimax Estimators Via Online LearningPradeep Ravikumar · Algorithmic Advances for Statistical Inference with Combinatorial Structure
• 31mQ-learning with Uniformly Bounded VarianceAdithya Devraj · Reinforcement Learning from Batch Data and Simulation
• 01h 00mMin-Max Optimization from a Dynamical Systems ViewpointPanayotis Mertikopoulos · Adversarial Approaches in Machine Learning 2022
• 55mOptimality and Approximation with Policy Gradient Methods in Markov Decision ProcessesAlekh Agarwal · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 46mSmooth Nonconvex Min-Max OptimizationMeisam Razaviyayn · Adversarial Approaches in Machine Learning 2022
• 31mStable Reinforcement Learning with Unbounded State SpaceQiaomin Xie · Reinforcement Learning from Batch Data and Simulation
• 35mThe Mean-Squared Error of Double Q-LearningR. Srikant · Reinforcement Learning from Batch Data and Simulation