Lessons from Unifying Simulation and RL Training
summary
Torc unifies object-sim and end-to-end neural-rendering RL on Ray and AnyScale, swapping GPU-drive sims for a CPU-vectorized stack to hit 2 billion steps in 40 minutes on six GPUs and peak 1M steps/sec on 8 H100 for mid-level training. A custom rollout engine called nerve lets them plug any Ray-first trainer, fuse perception and neural rendering, and run closed-loop replay buffers; the system yields >1 billion simulated miles per week and ~20x data-throughput gains.
AI-written summary · about
chapters (10)
- 0:00Introduction and speakers
- 1:41Autonomous trucking challenges
- 3:50Why reinforcement learning
- 5:59Scenario mining and training
- 7:06Mid-level representation RL
- 9:19Testing trained policies
- 17:25End-to-end neural loop
- 20:15Scaling simulation and training
- 23:45Modular system integration
- 25:51Achievements and future
related talks
28mScaling Multimodal Data and Reinforcement Learning with Ray
15mRollArt: Disaggregated Multi-Task Agentic RL Training at Scale
17mScaling Reinforcement Learning
29mScaling ML Pipelines for Autonomous Driving with Ray
37mA practical perspective on training reinforcement learning bots.
23mUsing RLLib in an Enterprise Scale Reinforcement Learning Solution
19mA 24x Speedup for Reinforcement Learning with RLlib + Ray
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training