Closing the Train-Inference Divide for Asynchronous RL at Scale
summary
Implements faithful replay of router decisions and per-token top-K/top-p support in SkyRL, capturing rollout metadata and reconstructing it exactly on trainer side. On a 30B model replay lowers router drift and preserves entropy; at production scale naive metadata causes 100+ GB arrays and CPU/network bottlenecks. Engineering fixes include lower-precision expert indices, sequence-packed binary encodings, Ray object-store tuning and PyTorch sparse tensors; mitigations for VLLM edge cases (nan / out-of-top-K ~1 per 100M tokens) restore stability and keep throughput competitive.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 1:05Async RL and model drift
- 3:56Initial implementation and small-scale results
- 5:20Scaling challenges with metadata
- 7:47Optimizing top P sampling
- 9:10Numerical edge cases at scale
- 10:56Real-world training results
- 12:21Key takeaways and conclusion
related talks
27mSkyRL: A Modular RL Library for LLM Agents
25mOptimizing Reinforcement Learning at Trillion-Parameter Scale
02h 42m[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents
18mMaximizing Luck in Reinforcement Learning
17mCrystalBox: Efficient Model-Agnostic Explanations for Deep RL Controllers
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training
17mRollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching
21mAdvanced Ray RLlib: New Features and New Possibilities