Hard Won Lessons from Building Effective AI Coding Agents
summary
Argues that frontier models outperform elaborate stacks: Gemini 3.0 immediately tops terminal bench leaderboards running on a stripped-down Terminus harness, showing capability beats scaffolding. Describes turning real coding logs into containerized RL environments (Docker, removed Git) via a qualification → archaeology → verifier pipeline, with verifiers tested as outcome-only "teak-kettle" checks. Reports reducing environment build time from ~16 hours to under 20 minutes per task and launches an open-source client bench of reproducible RL/eval environments for community SFT and RL.
AI-written summary · about
chapters (8)
- 0:00The bitter truth
- 2:03Rethinking agent design
- 3:43The real bottleneck
- 5:33Creating RL environments
- 7:38Building good verifiers
- 10:01Automating the process
- 11:11The truth nuke
- 12:29Announcing cline bench
related talks
33m(No) Future of coding and scripting in AI era
26mAl and Software: The Good, The Bad, The Scary
27mHow to Build AI Agents and Keep Your Sanity
26mCoding with AI
37mGenerative AI: Beyond (Just) Hype
30mMind the Gap - A Developer's Roadmap To Building AI Agents
41mThe High Stakes Security of AI
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability