I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
summary
Non-technical users see local SHAP values in a spreadsheet interface and feel they understand the model yet the illusion of explanatory depth appears: subjective understanding rises with SHAPTable prototype but then drops when tested. The study uses 40 moderated, 107 crowd workers and a loan default classifier with SHAP explanations. Moderated participants spend more time, score better than unmoderated ones, and calibration reveals missed feature interactions and overconfidence despite above‑chance objective performance.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:55XAI Is About an Al System's "Ability to Explain or Present in Understandable Terms to a Human"
- 1:56Local Explanations Leave it up to the User to Construct a Global Understanding of the Model
- 3:40Are Non-Technical Users of XAI Systems Based on Shapley Explanations Prone to an IOED?
- 5:03Mixed Methods Approach to Account for Rational and Heuristic Modes of Human Reasoning
- 6:27Participants in Both Settings Decreased their Subjective Understanding Throughout the Procedure
- 7:20Moderated Participants Were Less Confident and Gained a More Accurate Objective Understanding
- 8:11XAI Must Account for Cognitive Biases When Exposing Non-Technical Users to Explanations
related talks
19mMoving Forward with Explainable AI: Adoption Rate & Essential Dimensions for a Reliable Use
37mInteractive Explainable AI
33mBringing life and motion to AI Explainability
39mKeynote 3: How did you know? Explaining Black Box Model Predictions in Python
39mGuide towards algorithm explainability in machine learning
39mThe Search for Transparency and Accountability in the Age of...
45mAI-404: Explanation Not Found
41mIf We Want AI to be Interpretable, We Need to Measure Interpretability