Keynote: Can you trust your (large language) model?
summary
Compares concrete failures and fixes: a psychology replication snafu, CEXNet overfitting because of data leakage, and GPT‑4 solving Codeforces puzzles only when those puzzles were in its training set; finds at least 15% of ML studies have measurement errors, leaderboards (MMLU, HellaSwag) contain bad items, and the LM Arena rankings flip if 10% of voters are apathetic or adversarial. Shows prompt sensitivity up to 77% difference, warns of token scaling from 500B to 36T and dataset exhaustion by 2027, and recommends unit tests and human‑in‑the‑loop evaluation with LLM‑as‑judge and A/B testing.
AI-written summary · about
chapters (12)
- 0:00The problem of trust
- 1:14Lessons from psychology
- 5:09Errors in machine learning
- 7:30Reliability and validity
- 11:36Challenges in LLM assessment
- 15:24The myth of human benchmarks
- 21:25Flaws in LLM leaderboards
- 27:24Validity and scope issues
- 33:47Vibes-based evaluation
- 38:25The threat of data leakage
- 44:39Prompt sensitivity
- 47:51Building robust evaluation
related talks
54mKeynote: Can you trust your (large language) model?
01h 04mCan you trust your (large language) model?
01h 05mCan you trust your (large language) model?
43mBeyond the Hype: A Realistic Look at Large Language Models
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
31mChallenges and Considerations in Language Model Evaluation
30mLarge Language Models and Prompt Engineering for Developers
46mHow Large Language Models Are Reshaping the Cybersecurity Landscape