Optimizing Reinforcement Learning at Trillion-Parameter Scale
summary
Trillion-parameter MoE reasoning model with 384 experts and ~32.6B active parameters per token trains using LoRA adapters (rank 32 across all linear layers) so only the low-rank deltas are updated; Megatron-Bridge syncs those deltas to Megatron while vLLM and veRL handle rollout and inference. Merge vs bridge trade-offs: merge writes full weights (high bandwidth), bridge ships deltas (lower comms, needs LoRA serving). router replay R3 records logical top-K expert IDs during vLLM rollouts and replays them in Megatron to eliminate router-induced training–inference divergence.
AI-written summary · about
chapters (8)
- 0:00Trillion-parameter RL for LLMs
- 4:41Reinforcement learning after pre-training
- 9:20LoRA at trillion-parameter scale
- 12:07Training-inference mismatch in MoE
- 15:42Router replay for MoE RL
- 19:57RL system design principles
- 21:21Mint for RL infrastructure
- 23:10Resources and closing
related talks
31mVeRL: Extreme Optimization Practices for Ultra-Large-Scale MoE Models
14mClosing the Train-Inference Divide for Asynchronous RL at Scale
27mStageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference
39mAccelerating Open-Source RL and Agentic Inference with vLLM
46mArchitecting Efficient Agents with Semantic Routing and Mixture-of-Tools
20mEverything I Learned Training Frontier Small Models
49mThe Building Blocks of Today’s and Tomorrow’s Language Models
41mMeet DBRX, the New Standard for High-quality LLMs, and How...