What Do Models Still Suck At?
summary
Data come from Arena: 700 tracked text models, a 155‑question nonsense benchmark and 5.5 million votes across real user prompts. Models often accept nonsense instead of pushing back — about 50/50 for some families — and adding “more reasoning” sometimes makes answers worse. Overall dissatisfaction falls (top‑model rate roughly to 9%), but on an expert subset of ~40,000 prompts software requests show dissatisfaction dropping only from 23.5% → 13%, so some practical tasks still lag.
AI-written summary · about
chapters (7)
- 0:00Model limitations
- 1:11Nonsense question benchmarks
- 6:23Reasoning performance trends
- 9:12Arena dissatisfaction insights
- 13:04Expert category analysis
- 16:15Software and gaming trends
- 19:25Closing and future
related talks
31mChallenges and Considerations in Language Model Evaluation
54mKeynote: Can you trust your (large language) model?
10mWhat goes into the making of LLMs
54mKeynote: Can you trust your (large language) model?
01h 04mCan you trust your (large language) model?
16mNavigating the Uncharted: Ensuring Prompt Quality in the Age of Language Models
01h 05mCan you trust your (large language) model?
16mBenchmarks Are Memes: How What We Measure Shapes AI—and Us