Panel Discussion: Beyond the Leaderboard...
summary
Companies build realistic tool-driven environments to test assistants on German car manuals and safety constraints, and a platform called ProfitArena predicts future events by asking LMs to forecast markets and elections (large-scale election calls prove reasonably accurate). Public benchmarks show widespread label errors and nonrepresentative traces, so teams stop using static leaderboards and abolish perplexity in favor of per-token accuracy, add fill‑in‑the‑middle coding tests, and run async event‑driven agent workloads instead.
AI-written summary · about
chapters (13)
- 0:00Panel discussion intro
- 1:05Panelists' expertise
- 3:10Academic vs real-world evals
- 6:19Leaderboards stifling innovation?
- 9:50History of AI evals
- 11:36Profit arena applications
- 13:22Open vs closed evals
- 16:57Replacing benchmarks
- 21:15Perplexity and coding evals critique
- 24:01Designing meaningful evals
- 27:31What not to do when creating evals
- 28:36Non-standardized domain evals
- 29:39Q&A
related talks
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
44mPanel Discussion | The AI Revolution in Testing: Opportunities and Challenges
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
45mPanel Talk
46mEnhancing Agent Reliability Through Evaluation-Driven Development
54mKeynote: Can you trust your (large language) model?
19mAI's Infinite Game
01h 05mCan you trust your (large language) model?