RL for Autonomous Coding
summary
Argues that scaling LLMs with post-training open-weight LLMs plus reinforcement learning measurably raises autonomous coding success, citing PaLM-scale models (540B) and verified-code benchmarks that reach roughly 80% BenchVerified with heavy sampling. Uses inference-time tricks like self-consistency/majority voting and iterative revision but shows they need thousands of samples, so RL with automated verifiers (unit tests, execution feedback) is necessary. Notes system costs: RL requires multiple model copies (PPO/DPO variants, four vs three copies) and risks reward hacking, making engineering at scale the main barrier.
AI-written summary · about
chapters (9)
- 0:00Introduction to LLMs and Scaling Laws
- 1:41Emergent Behavior in LLMs
- 4:00Reinforcement Learning from Human Feedback (RLHF)
- 6:11Inference-Time Scaling and Verification
- 10:33Challenges with Inference-Time Scaling
- 11:16The Next Frontier: Reinforcement Learning for Correct Generation
- 13:20Challenges in Scaling RL
- 14:58Autonomous Coding as a Prime Domain for RL
- 15:53Reflection.ai's Mission
related talks
39mAccelerating Open-Source RL and Agentic Inference with vLLM
19mLLMs reasoning and agentic capabilities over time
26mScaling Agentic Intelligence from Pre-Training to RL
35mLLM-Guided Reinforcement Learning for Mastery Learning: Large-Scale Field Evidence
27mBuilding Mini Minds Our Adventures in Teaching Small LLMs to Reason
15mOpenAssistant's Opensource LLMs Saga
24mverl: An Open Source Large Scale LLM RL Framework for Agentic Tasks
17mRLM: Recursive Language Models for Large Codebases