Chatbot Arena: UC Berkeley's Open AI Evaluation Platform
summary
An open platform for human-preference LLM evaluation collects millions of pairwise votes and runs live side-by-side comparisons, reporting 1.7 million user votes (uploader) and about 2 million pairwise votes in analysis, ranking a leaderboard of 140+ models from community traffic. The site serves models with an open-source multi-model serving stack, releases a 1M-conversation dataset plus 100k human preferences, detects noisy or malicious raters, and produces statistically calibrated rankings that update in days.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:34About Me
- 1:59AI Progress
- 3:08Benchmark Leakage
- 5:24Challenges
- 6:40What can we do
- 7:47How we build it
- 11:25Progress
- 13:29Results
- 17:34Detection
- 19:56Open Source
- 21:41Collaboration
- 25:33Community Driven
related talks
27mChatbot Arena: An Open Crowdsourced Platform for Human Feedback on LLMs
26mAI Showdown: Open Source Tools for LLM Face-Offs
27mMeasuring Intelligence in the Wild: Advances in Preference-Based Evaluation
20mAgentic Evaluations at Scale, For Everybody
33mEvaluating Your AI Agent
19mBuilding an Agentic Platform
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
14mKeynote: Building AI with Open Source & Hugging Face