Toxic Trust: The Psychology of Automation Bias in Generative AI
summary
Automation bias causes real-world harms: an Austrian hiring score AMS scoring deducts points for women and immigrants and a Dutch fraud detector helps collapse a government, a pharmacist robot dispenses 38.5 pills overdose and nearly kills a child, GPS plus trust repeatedly sends drivers into water, and developer tools mislead — GitHub Copilot produces bubble sort and wrong fixes — because people accept machine output under time pressure. A practical counterexample is a reproducible GitHub Actions pipeline that generates conference abstracts, surfaces model metrics and flags GPTy phrasing — GitHub Actions abstract tool shows model metrics — and it works in production.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 5:59Automation bias basics
- 10:31Model vs automation bias
- 13:22Government failures
- 17:53GPS and spell check examples
- 21:03Medical error case
- 24:34Google AI mistakes
- 29:48Software testing errors
- 33:45Why humans suffer from automation bias
- 39:47Why solutions fail
- 42:16Reducing automation bias
- 47:11Human review principle
- 50:40Automated abstract tool
- 55:57Conclusion and key takeaways
related talks
44mToxic Trust: The Psychology of Automation Bias in Generative AI
12mThe Comfort Trap: Getting Too Used to AI
33mHuman Factors in the Age of AI Ops: Re-Engineering Trust between Humans and...
34mTrust in the Age of AI Agents | Building Trustworthy AI
49mMagic of Automation and Everyday Chores
34mScaling Trust, Not Automation - Rethinking Quality for the AI Era
01h 07mUnpoisoned Fruit: Seeding Trust into a Growing World of Algorithmic Warfare
46mCan you trust your AI?