Evals Are Broken, Use Them Anyway
summary
Benchmark numbers are not gospel and vibes are not a system: a run starts at 43% on Terminal Bench and most gains come from container CPU and memory tweaks, raised timeouts, and prompt engineering that helps Anthropic families but does not transfer to Codex or Gemini. After a run, failures get portfolio allocated by sending an agent through failure traces to find small levers. Three zones emerge: obvious bugs, nuance improvements, and overfitting.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:20Eval misconceptions
- 3:24Stages of using evals
- 4:27Interpreting eval results
- 6:34Improving AI agents
- 10:24Custom evals
- 13:10Eval failures
- 15:58Eval improvement zones
- 17:01Call to action
related talks
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
08mHow to Improve your Vibe Coding
15mEvals Are Not Unit Tests
06mWhy should anyone care about Evals?
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
20mFive hard earned lessons about Evals
08mProduction Evals For Agentic AI Systems
16mTurning Fails into Features: Zapier’s Hard-Won Eval Lessons