Gradient-Descent for Randomized Controllers under Partial Observability

on YouTube ↗28m

" summary

Transforms partially observable controllers into a parametric Markov chain and computes exact derivatives by building a derived weighted automaton so gradients equal expected rewards on that automaton, then applies multiple gradient-descent variants (plain, momentum, Nesterov, adaptive, momentum sign) with minibatch updates (32) and three region-restriction schemes (projection, log-barrier, sigmoid). Projection and momentum-sign win empirically, often beating QCQP and particle-swarm optimization by about an order of magnitude on benchmark POMDP-derived pMCs.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 0:30Partial Observability
3. 3:47Parametric Markov Chain
4. 8:21GradientDescent
5. 12:09GradientDescent Methods
6. 13:54Restriction Methods
7. 15:26Experimental Validation
8. 15:48Model Checker
9. 18:22Feasibility Synthesis
10. 20:04Particle Swarm Optimization
11. 20:33Summary
12. 21:08Questions
13. 22:42Conclusion

" related talks

• 45mPolicy Gradient: Optimal Estimation, Convergence, and Generalization beyond Cumulative RewardsMengdi Wang · Adversarial Approaches in Machine Learning 2022
• 01h 00mMin-Max Optimization from a Dynamical Systems ViewpointPanayotis Mertikopoulos · Adversarial Approaches in Machine Learning 2022
• 31mLearning to Control Unknown SystemsRahul Jain · Societal Networks 2018
• 35mOn the Global Convergence and Approximation Benefits of Policy Gradient MethodsDaniel Russo · Mathematics of Online Decision Making 2020
• 55mOptimality and Approximation with Policy Gradient Methods in Markov Decision ProcessesAlekh Agarwal · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 41mCompetitive Algorithms for Online ControlYisong Yue · Mathematics of Online Decision Making 2020
• 53mOff-policy Policy OptimizationDale Schuurmans · Emerging Challenges in Deep Learning (Simons Institute workshop, 2019)
• 42mReinforcement Learning using Generative Models for Continuous State and Action Space SystemsRahul Jain · Reinforcement Learning from Batch Data and Simulation