Optimizing RLHF Training for Large Language Models with Stage Fusion
summary
RLHFuse claims up to 3.7× end-to-end RLHF training throughput by splitting tasks into sample-level subtasks and microbatch subtasks. It overlaps generation and inference via a data-aware interstage fusion that migrates completed samples to designated GPUs to avoid long-tail decoding stalls, and it fuses actor and critic training with a model-aware intrastage fused pipeline schedule that handles heterogeneous model sizes. Evaluated on 13B–65B reward models on a 256 H800-GPU cluster; interstage code is open source.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:47RLHF training basics
- 3:14Single RLHF iteration workflow
- 4:40Challenges in RLHF
- 6:26Breaking tasks into subtasks
- 7:29Data-aware fusion technique
- 9:13Model-aware fusion technique
- 11:21Experimental results
- 12:27Summary and open source
- 12:49Q&A
related talks
15mPUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context...
17mRLinf: Flexible and Efficient Large-Scale Reinforcement Learning...
17mScaling Reinforcement Learning
17mReinforcement Learning with Human Feedback in Distributed Systems
15mRollArt: Disaggregated Multi-Task Agentic RL Training at Scale
01h 01mFederated Learning in the Generative AI Era
20mEverything I Learned Training Frontier Small Models
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training