The Art & Science of Benchmarking Agents
summary
Benchmarks need concrete signals: ARC AGI 3 with every task human‑solvable and frontier models under 1%, Snorkel reviews over 120 applications and funds a $3M Open Benchmarks Grants. Useful benchmarks show validated task quality, distributional diversity, unsaturated model headroom and robust evals (cost, latency, traces). The art is having a thesis—Terminal Bench bet on the CLI—shipping research roadmaps, tooling that makes contribution and reproduction trivial, and benchmarks that produce new, measurable engineering practices.
AI-written summary · about
chapters (7)
- 0:00Benchmarking agents intro
- 2:01Evaluation gap in agents
- 3:44Open benchmarks shape field
- 6:29Building effective benchmarks
- 13:08Frontier-shaping benchmarks
- 18:06Next-gen benchmark dimensions
- 22:39Closing and grants
related talks
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
40mBeyond Static Benchmarks: Chaos Based Agent...
31mBuilding High-Quality & Trustworthy FMware
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
16mMaking Codebases Agent Ready
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability