Hallucinations: Harvesting Uncertainty
summary
Claims that large models expose a usable internal uncertainty signal readable before generation and token by token, and that token probabilities are too noisy so the right place to probe is latent space features. Implements a simple logistic regression probe trained on datasets like TriviaQA to predict correctness, yielding span‑level hallucination detection and a token confidence head. Uses this for model/human escalation, abstention, and speculative decoding, delivering a 60–85% speculative decoding speedup and deployed by Deepseek and Liquid AI.
AI-written summary · about
chapters (7)
- 0:00AI trust and hallucinations
- 1:56Why hallucinations happen
- 3:21Tactics to reduce AI errors
- 4:26Detecting model uncertainty
- 6:35Hallucination detection in practice
- 8:00Speeding up AI with uncertainty
- 9:05Unchecked AI confidence risks
related talks
32mWhy do LLMs hallucinate?
34mDisincentivizing Hallucination
41mTruth Checker: Generative Large Language Models and Hallucinations
37mAI Hallucination Explained 🤯 Why AI Gets Things Wrong
01h 06mWhen calibration goes awry: hallucination in language models
29mLies, damned lies and large language models
01h 15mUnderstanding (a bit about) hallucinations in Generative AI
39mLLMs and Hallucinations: Understand, Measure, Control