Benchmarks Are Memes: How What We Measure Shapes AI—and Us
summary
Benchmarks act like cultural memes shaping model behavior: Every Inc. builds an AI Diplomacy benchmark and runs 18 models against each other, finding Gemini 25 Pro hits 16 centers early while model 03 consistently schemes to win; Claude Opus proves overly trusting, Llama For Maverick excels at social persuasion, and DeepSeek R1 nearly wins after a recent update. Other concrete findings: video counting (1–10) remains unsolved and Pokemon gameplay is midway to saturation.
AI-written summary · about
chapters (9)
- 0:00Benchmarks as memes
- 1:12The problem with saturation
- 2:31The benchmark life cycle
- 4:24Empowering the community
- 5:14Responsibility and design
- 6:18Qualities of great benchmarks
- 7:38Introducing AI diplomacy
- 8:27Lessons from AI gameplay
- 14:14Human agency in AI's future
related talks
44mKeynote: The neglected art of a relevant benchmark and why you need one
30mPanel Discussion: Are We Measuring Intelligence or Just Benchmarks?
19mAI's Infinite Game
19mApples, Oranges, and ML Models: Model Validation vs Benchmarking
29mBenchmarking GenAI Like a Pro: Scaling Experiments, Predicting Performance, and Keeping Your Sanity
31mBuilding High-Quality & Trustworthy FMware
31mPanel Discussion: Beyond the Leaderboard...
11mWhy AI is the ultimate mirror of human ambition: A journey into the heart of machine thought