Measuring Intelligence Summit 2025
2025-10-21
San Francisco, United States
who it's for
evaluation researchers, language-model researchers, reinforcement learning researchers, machine learning engineers, infrastructure engineers, product managers at AI companies, startup founders and VCs
topics
HELM reproducible evaluation framework; weak verifier ensemble distillation; 400M distilled verifier models; preference-based live evaluation (LMArena); pairwise Bradley-Terry comparison modeling; scaling agentic RL from pretraining to RL; chain-of-thought and long-context (>100k tokens) methods; verifiable reward signals (calculator/unit test checks); dynamic evaluation and private holdouts; production ML stacks and Meta silicon; decentralized and open RL environments; per-token accuracy and fill-in-the-middle coding tests
| title | author | length |
|---|---|---|
| Keynote: Framing the Frontier of Machine Intelligence | Joe Spisak | 04m |
Agenda outlines concrete evaluation and infra work: morning focuses on Helm evaluation and benchmarked results plus the Weaver weak verifiers RL project; a… | ||
| Keynote: Noam Brown, Research Scientist at OpenAI in Conversation with Joe Spisak | Noam Brown, Joe Spisak | 31m |
Leads reasoning work at OpenAI and traces a path from superhuman poker AIs to Cicero diplomacy agent that placed second in its league yet was preferred by… | ||
| Weaver: Shrinking the Generation-Verification Gap with Weak Verifiers | Jon Saad-Falcon | 26m |
Weaver aggregates many small, noisy verifiers (reward models and LM judges) without labels to close the generation‑verification gap, matching o3‑mini on… | ||
| Holistic Evaluation of Language Models (HELM) | Yifan Mai | 29m |
HELM is an open-source Python framework plus public leaderboards that run reproducible model evaluations; it executes 42 benchmarks on 30 models using roughly… | ||
| Panel Discussion: Are We Measuring Intelligence or Just Benchmarks? | Sara Hooker, Vivienne Zhang, Nathan Habib | 30m |
Argues that static public benchmarks quickly saturate or leak into training, cites reproducibility variance of reproducibility delta >10% across… | ||
| Scaling Agentic Intelligence from Pre-Training to RL | Aakanksha Chowdery | 26m |
Covers how scaling from pre‑training to reinforcement learning uses long chain of thought examples plus RL with verifiable rewards (calculator checks, unit… | ||
| Measuring Intelligence in the Wild: Advances in Preference-Based Evaluation | Anastasios Angelopolous | 27m |
LMArena is a live benchmark with tens of millions of users and hundreds of millions of prompts, exposing 700+ models across 150+ countries and producing 250… | ||
| Panel Discussion: Beyond the Leaderboard... | Jeremy Howard, Rebecca Qian, Haifeng Xu, Shishir G Patil | 31m |
Companies build realistic tool-driven environments to test assistants on German car manuals and safety constraints, and a platform called ProfitArena predicts… | ||
| nothing matches | ||