Measuring Intelligence in the Wild: Advances in Preference-Based Evaluation
summary
LMArena is a live benchmark with tens of millions of users and hundreds of millions of prompts, exposing 700+ models across 150+ countries and producing 250 million conversations across 125+ languages. It uses blind randomized pairwise comparisons and Bradley‑Terry logistic regression with style and sentiment controls, trains a prompt‑conditional model (7B shown) called Prompt-to-Leaderboard (P2L) to output per‑prompt Bradley‑Terry vectors, deploys those vectors for cost/latency optimal routing via a linear program, and reports significant PTO improvements.
AI-written summary · about
chapters (14)
- 0:00Alamarina mission
- 1:03Live human evaluations
- 2:28Text and image platforms demo
- 3:52Platform scale and impact
- 5:57Live vs static benchmarks
- 9:27Granular analytics challenges
- 10:51Leaderboards with Bradley-Terry
- 13:18Beyond average leaderboards
- 16:28Prompt-specific leaderboards
- 19:17Prompt model scaling laws
- 20:21Routing cost-performance trade-offs
- 21:24Preference decomposition
- 24:33Causal inference directions
- 25:14Closing remarks
related talks
38mPrompt Engineering ⚙️ - Addressing the sensitivity of Large language models
32mChatbot Arena: UC Berkeley's Open AI Evaluation Platform
31mChallenges and Considerations in Language Model Evaluation
26mHow To Win Friends & Influence LLMs (with Prompt Engineering)
16mNavigating the Uncharted: Ensuring Prompt Quality in the Age of Language Models
11mThe Unreasonable Effectiveness of Prompt Learning
26mMastering Large Language Models: Prompt Engineering for Precision & Performance
42mAdvancements in Evaluating Large Language Model Applications