Can you trust your (large language) model?
summary
Retells the Amy Cuddy power‑posing replication failure to show how plausible results can be wrong, then walks through concrete LLM failures: benchmark glitches in MMLU and HellaSwag, leaked training data from Codeforces causing GPT‑4 to ace puzzles, Llama 3 trained on 15 trillion tokens, and extreme prompt sensitivity that swings accuracy by up to 77%. Recommends engineering fixes: unit tests, logging traces, human evaluation and A/B tests, and warns about data leakage and multiple‑choice versus generative evaluation mismatches.
AI-written summary · about
chapters (16)
- 0:00Trusting large language models
- 3:36Power posing controversy
- 9:56Replication crisis in ML
- 13:53Reliability in model assessment
- 15:40Data leakage impact
- 17:47Black-box model validity
- 21:00Human benchmarks for LLMs
- 26:23LLM benchmark problems
- 31:42Leaderboard limitations
- 33:30Multiple-choice vs free-form
- 35:16Reasoning benchmark issues
- 39:53Chatbot Arena challenges
- 43:04Code generation leakage
- 47:41Prompt sensitivity issues
- 51:12Engineering LLM evaluation
- 57:36Q&A
related talks
54mKeynote: Can you trust your (large language) model?
54mKeynote: Can you trust your (large language) model?
01h 05mCan you trust your (large language) model?
43mBeyond the Hype: A Realistic Look at Large Language Models
29mLies, damned lies and large language models
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
46mHow Large Language Models Are Reshaping the Cybersecurity Landscape
44mI Abuse Your LLM Live with 10 Security Vulnerabilities