ByteDance's Platform for Reinforcement Learning from Human Feedback
summary
ByteDance describes a production RLHF platform that runs LLM trajectory generation and training on Ray while keeping model-parallel LLM compute inside SPMD processes, claiming order-of-magnitude less idle GPU time from avoiding full-weight transfers. Core pieces are hybrid controller (single-process orchestration + SPMD worker groups), worker group APIs that shard/gather batches, and a weight resharing manager that remaps FSDP <-> tensor-parallel layouts; reported wins over DeepSpeed Chat / OpenRLHF and a Eurosys 2025 paper.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:07Alignment
- 2:37Language Model Generation
- 4:08Example
- 6:31Reward Model
- 7:27Model Based RL
- 12:50Single Controller vs M Controller
- 16:27Key Observations
- 18:05Hybrid Controller
- 19:31Worker Group
- 23:23IRF Framework
- 26:12Weight Transfer
related talks
29mRay:A Distributed Computing Platform for Reinforcement Learning
17mReinforcement Learning with Human Feedback in Distributed Systems
25mVerl: A Flexible and Efficient RL Framework for LLMs
17mScaling Reinforcement Learning
17mRLinf: Flexible and Efficient Large-Scale Reinforcement Learning...
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training
28mScaling Multimodal Data and Reinforcement Learning with Ray
19mML training acceleration with heterogeneous resources in Bytedance