Stop Evaluating Models Like It's the 50s
summary
Claim: applying Item Response Theory to model evals yields per-item difficulty (B), discrimination (A) and per-model theta with real error bars instead of raw accuracy. Using real epoch.ai data and examples like GPT 5.5, Gemini 3 Pro and Claude, IRT finds mislabeled items, detects contamination and produces adaptive testing that shrinks a 484‑item benchmark to ~97 items — a 5x reduction while keeping ~99% rank correlation — and creates residual fingerprints that reveal leakage and related models.
AI-written summary · about
chapters (12)
- 0:00Model evaluation psychology upgrade
- 1:47Benchmark question equality problems
- 3:12Item response theory for LLMs
- 6:42IRT outperforms raw accuracy
- 8:26Auditing benchmarks with IRT
- 10:33Shrinking benchmarks without signal loss
- 13:19Fewer questions better results
- 14:03Detecting benchmark leaks
- 16:52Adaptive testing for benchmarks
- 18:39Biased items in evaluations
- 20:03Model fingerprinting with residuals
- 22:09Psychometrics in AI future
related talks
15mEvals Are Not Unit Tests
32mThe Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
22mCracking the Code: Deciphering Evaluation Essentials for RAG
31mAccuracy and Interpretability Through the Lens of Human-AI Teaming
49mEvals 101
50mThe ART of Probable: Test with AI, Atomic Red Team, and Threat Metrics
40mHow [not] to evaluate your RAG