Let LLMs Wander: Engineering RL Environments
summary
Builds a Tic-Tac-Toe RL environment with the open-source Verifiers library, using deterministic Python verifiers, XML parsing for moves, and rewards of 1/0.5/0 with a -0.1 penalty for invalid moves; trains via group-based GRPO style updates (group size ~8, batch 256), varying opponent randomness from 20–70% then 0–25%. Uses Primer RL for warm-up SFT, tests models like LFM2 and GPT-5 mini, and reports a small model improving from frequent losses to nearly always drawing vs optimal and matching or beating GPT-5 mini on this task.
AI-written summary · about
chapters (20)
- 0:00Introduction and agenda
- 1:31Reinforcement learning basics
- 3:18LLM training paradigms
- 5:51RL with verifiable rewards
- 8:20Mapping RL to LLMs
- 10:07Building RL environments
- 11:35Simple Verifiers environment
- 14:06Multi-turn environments
- 15:10Tool environments
- 17:21Tic-Tac-Toe experiment
- 18:26Initial implementation
- 21:18Improving the environment
- 23:09Handling invalid moves
- 24:32Evaluating existing models
- 25:37Supervised fine-tuning warm-up
- 27:02GRPO reinforcement learning
- 29:55Training and results
- 31:38Adjusting exploration
- 33:10Key takeaways and future
- 35:19Q&A session
related talks
41mLet LLMs Wander: Engineering RL Environments
19mTraining Agentic Reasoners
25mDesigning and Building Custom Reinforcement Learning Environments for Fine-tuning LLMs
15mAgentic Reasonings with Reinforcement Learning
27mBuilding Mini Minds Our Adventures in Teaching Small LLMs to Reason
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
19mRL Environments at Scale
44mWhy RL Environments Are All You Need for Building Agents