Interpretability - now what?
summary
Shows that common saliency maps barely change when network weights are randomized and fail simple sanity checks, so a benchmark with controlled object placement and metrics (BEAM dataset, input-dependence/input-independence and model-contrast scores) is built. Proposes TCAV using concept activation vectors: train linear separators on 15–33 example concepts, take directional derivatives to score importance, passes synthetic and medical tests and surfaces noisy labels in diabetic retinopathy models.
AI-written summary · about
chapters (21)
- 0:00Intro
- 3:01My goal interpretability
- 4:37NON-goals
- 6:58Investigating
- 8:14Sanity check question.
- 15:46Benchmarking interpretability methods (BIM)
- 16:57Three metrics for measuring false positives
- 19:10Model Contrast Score (MCS)
- 25:19Defining concept activation vector (CAV) Inputs
- 27:19TCAV core idea: Derivative with CAV to get prediction sensitivity
- 29:25Quantitative validation: Guarding against spurious CAV
- 31:12Recap TCAV: Testing with Concept Activation Vectors
- 32:16Sanity check experiment setup
- 34:50Human subject experiment: Can saliency maps communicate the same information?
- 36:45TCAV in Two widely used image prediction models
- 39:23Collect human doctor's knowledge
- 39:46TCAV for Diabetic Retinopathy
- 40:51Summary: Testing with Concept Activation Vectors
- 41:08Responses from inside of academia
- 42:48Limitations of TCAV
- 44:02Things to keep in mind during our journey
related talks
21mInterpretability For Everyone
01h 00mData, visualization, and designing AI
09mQuantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark
41mModels Schm-odels: Why you Should Care about Data-Centric AI
19mHow to look at your data
35mDeep Learning Model Interpretability for Computer Vision based Models
38mWhere are we looking? Predicting human gaze using deep networks.
34mInterpretability and Algorithmic Fairness