MSRL: Distributed Reinforcement Learning with Dataflow Fragments
summary
Claims a new fragmented dataflow graph abstraction that splits an RL training loop into independent fragments mapped to CPU, CUDA kernels or DNN operator graphs, and swapped across devices by a distribution policy. Implements MSRL and shows concrete wins: about 3× faster on PPO, 2× on A3C versus a baseline, and up to 58x speedup over a sequential baseline with 32 agents/32 GPUs; 64-agent runs reach ~7600 Hz input throughput. Code is on GitHub.
AI-written summary · about
chapters (9)
- 0:00MSRL and RL challenges
- 6:27Efficient RL training challenges
- 8:34MSRL solution with dataflow fragments
- 13:14Fragmented dataflow graphs
- 15:20Distribution policies for performance
- 18:28MSRL system architecture
- 19:56Experimental results and gains
- 23:29Scalability and throughput
- 24:13Conclusion and future
related talks
17mRLinf: Flexible and Efficient Large-Scale Reinforcement Learning...
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training
15mRollArt: Disaggregated Multi-Task Agentic RL Training at Scale
25mVerl: A Flexible and Efficient RL Framework for LLMs
16mDistRS: Disaggregated Reward Service for RLVR with Batch-Level Constraint
17mScaling Reinforcement Learning
17mReinforcement Learning with Human Feedback in Distributed Systems
14mRLlib: Scalable RL for TensorFlow, PyTorch, and Beyond