Drowzee: Metamorphic Testing for Fact-conflicting Hallucination Detection in Large(…)
summary
Drowzee is a logic-programming–driven metamorphic tester that crawls Wikipedia to build a KB, applies Prolog rules (negation, symmetric, inverse, transitive, composite) to derive 7,200 Q&A cases across nine domains, then prompts six LLMs and flags fact-conflicting hallucinations using edge and node vector semantic oracles. Results show hallucination rates from about 24.7% to 59.8% (GPT‑4 ≈25%, ChatGPT ≈42%), expose temporal and OOD failures, and allow small-scale model edits (<1,000 facts) that reduce some errors.
AI-written summary · about
chapters (12)
- 0:00Drowzee framework intro
- 1:28Fact-conflicting hallucinations
- 3:17Logical programming for QA
- 5:05Enriched facts with Prolog
- 7:12Benchmark construction
- 7:54Hallucinations via metamorphic testing
- 9:19Effectiveness on LLMs
- 12:10Hallucination response types
- 13:56Drowzee vs benchmarks
- 15:18Ablation on inference rules
- 15:39Summary and future work
- 17:27Q&A
related talks
41mTruth Checker: Generative Large Language Models and Hallucinations
34mDisincentivizing Hallucination
32mWhy do LLMs hallucinate?
10mHallucinations: Harvesting Uncertainty
26mReducing Hallucinations and Evaluating LLMs for Production
15mReducing Hallucinations in Conversational Systems
29mLies, damned lies and large language models
37mAI Hallucination Explained 🤯 Why AI Gets Things Wrong