Panel Discussion: Are We Measuring Intelligence or Just Benchmarks?
summary
Argues that static public benchmarks quickly saturate or leak into training, cites reproducibility variance of reproducibility delta >10% across parsing/extraction tools, and points to SweepBench rising from single digits to ~70% (an 82% result is from a 1,000-agent network by Blitzy and not compute‑calibrated). Recommends private holdouts, dynamic evals (algorithmic regeneration like GSM8K+ or EFAL), standardized evaluation tooling (Helm/OpenLM), and better portability so enterprises can curate representative benchmarks and avoid overfitting.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:47Biggest misses in measuring intelligence
- 5:21Benchmarks vs real-world gaps
- 8:32Effective benchmark examples
- 10:20Root causes of benchmark failures
- 13:14Dynamic benchmarks and holdouts
- 15:44Incentivizing private data
- 17:31Overcoming benchmark inertia
- 19:57Adaptability and reasoning gaps
- 24:33Tools for adaptable systems
- 26:21Tactical benchmarking improvements
- 29:56Closing remarks
related talks
16mBenchmarks Are Memes: How What We Measure Shapes AI—and Us
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
31mBuilding High-Quality & Trustworthy FMware
01h 05mCan you trust your (large language) model?
25mReproducibility in Embedding Benchmarks
32mReproducibility in Embedding Benchmarks
01h 07mDiscussion Session