Can you trust your (large language) model?
summary
Argues that public benchmarks and human exams mislead: scores can reflect memorized training data not reasoning, with GPT‑4 solving Codeforces puzzles released during its training but failing post‑training ones, and leaderboards containing benchmarks with ~36% broken HellaSwag items and dozens of leaked QA sets. Shows massive training growth (500B to 15T tokens) and a finite ~300T usable token estimate, cites prompt sensitivity causing a 77% performance swing, data leakage confirmed, and recommends unit tests + human-in-loop assessments that proved practical on a real estate assistant.
AI-written summary · about
chapters (11)
- 0:00Trusting large language models
- 1:31Measuring model trust scientifically
- 8:28Reliability in model assessment
- 15:33Human-designed benchmarks fail for LLMs
- 22:38Unreliability of LLM leaderboards
- 28:16Answer formats affect LLM performance
- 32:55Limits of LLM problem-solving
- 41:45Data leakage in LLM training
- 47:50Assessing LLMs for your use case
- 53:51Can you trust your LLM?
- 1:00:17Q&A
related talks
54mKeynote: Can you trust your (large language) model?
01h 04mCan you trust your (large language) model?
54mKeynote: Can you trust your (large language) model?
43mBeyond the Hype: A Realistic Look at Large Language Models
56mSeparating fact from fiction in a world of AI fairytales
01h 00mInside GPT – Large Language Models Demystified
46mConversations with ChatGPT: illusion or reality?
30mLarge Language Models and Prompt Engineering for Developers