~/talk

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win

on YouTube 11m
Talkat AI Engineer World's Fair 2025 · 2025-06-03..06-05· [large-language-models, ai-assisted-development]

speaker

name:Darius Emrani
links:none on file

resources

video:https://youtu.be/EnT4Wej5M5k
slides:none on file

summary

Benchmarks control billions and the talk exposes three concrete cheat codes: Cherry-picking comparisons with xAI's Grok‑3 graphs, Buying privileged access via OpenAI funding FrontierMath, and Optimizing for style over substance as Meta submitted 27 Llama‑4 variants to LM Arena. Claims include 39% of score variance explained by style, consensus‑64 hiding cost tradeoffs, and cases where charming but wrong outputs beat correct ones. The proposed remedy is a 5‑step, production‑focused evaluation loop and open transparent test sets.

AI-written summary · about

chapters (4)

  1. 0:00Intro
  2. 2:09Tricks to win
  3. 7:01Fix public metrics
  4. 8:40How to win

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.