Training Agentic Reasoners
summary
Argues that RL at scale reliably improves multi-turn tool-using agents and shows a concrete pipeline: SFT warm-up, then RL with GRPO vs PPO tradeoffs for advantage estimation to avoid reward hacking. Releases a pip-installable verifiers toolkit that treats environments as harnesses and policies as LM APIs, and demonstrates a 7B model trained into a Wordle agent that solves games efficiently. Covers rewards, rubric-based LM judges, compute-for-few-GPUs engineering, and where evaluations still break.
AI-written summary · about
chapters (8)
- 0:00Introduction to the idea that reasoning and agents are similar.
- 1:05The growing effectiveness of Reinforcement Learning (RL) in AI.
- 3:04The complexities and challenges of implementing RL.
- 4:41The connection between popular AI products (agents) and RL fine-tuning.
- 7:18The core process of Reinforcement Learning.
- 10:21The importance of tools and real-world tasks for agents.
- 12:13The problem of "reward hacking" and how to design better evaluations.
- 14:51Future directions for agentic systems and a practical toolkit for implementation.
related talks
15mAgentic Reasonings with Reinforcement Learning
41mLet LLMs Wander: Engineering RL Environments
39mLet LLMs Wander: Engineering RL Environments
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
26mCan Reinforcement Learning Lead to AGI?
26mScaling Agentic Intelligence from Pre-Training to RL
36mSkyRL: Democratizing RL Training for Researchers
45mFrom Next Token Prediction to Reasoning and Beyond