Miles: Enterprise-Facing Agentic RL Framework
summary
Miles is an open-source PyTorch + Megatron + Ray stack that runs fully async agentic RL at frontier scale: a GM‑5.2 reference run hits ~263s median per step (≈4.5 min), 96% rollout cache hit, 1.7 step lag, and reward growth 0.44 → 0.56. Implements RDMA/RDT GPU→GPU weight sync, delta P2P updates, token middleware (token in/out session server) and routing replay to avoid MoE flips, saving tens of GBs HBM via CPU/NVMe offload and working end-to-end.
AI-written summary · about
chapters (9)
- 0:00Introduction to Miles
- 1:39The reference run
- 3:01The RL loop architecture
- 6:10Fully asynchronous RL
- 9:09Correctness and observability
- 13:33Training backend and efficiency
- 15:07Weight sync with Ray
- 24:29Roadmap and takeaways
- 28:49Closing
related talks
29mScaling ML Pipelines for Autonomous Driving with Ray
21mAdvanced Ray RLlib: New Features and New Possibilities
14mRLlib: Scalable RL for TensorFlow, PyTorch, and Beyond
06mBuilding Frontier AI Infra: SGLang and Miles
28mScaling Multimodal Data and Reinforcement Learning with Ray
19mNeural MMO: Building a Massively Multiagent Research Platform with Ray and RLlib
15mRollArt: Disaggregated Multi-Task Agentic RL Training at Scale
25mVerl: A Flexible and Efficient RL Framework for LLMs