What is Your Data Worth? Equitable Data Valuation in Machine Learning
summary
Defines data Shapley value as the marginal contribution of each training point averaged over all subsets and shows it outperforms the common leave-one-out score. Uses permutation-based estimates during SGD (cost similar to training) to compute values, finds negative Shapley points often indicate mislabeled or distribution-shifted records (e.g., Nottingham cohort hurts UK Biobank lung‑cancer accuracy), shows removing negative Shapley points or reweighting by Shapley yields double‑digit accuracy gains on held‑out deployment sets, and warns about statistical uncertainty, manipulability, and task dependence.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:33If data is fuel, then we need to measure its value
- 3:32Data value in the context of ML
- 4:27Ingredients of Data Value in ML
- 6:04Leave One Out Method
- 9:43Desirable properties
- 14:56Data Shapley Value
- 15:30Applications of Data Shapley
- 17:52UK Biobank Lung Cancer prediction
- 21:11Removing low value data improves prediction
- 23:05Adding high value data improves prediction
- 25:13Negative Shapley identifies mislabeled data
- 27:57Domain adaptation: face recognition
- 30:42Dermatology classification
- 31:47Clinical notes NLP
- 32:14Efficiently approximating data Shapley
- 33:53New frontiers of data valuation
- 36:30Discussion
related talks
26mShparkley: Scaling Shapley values with Spark
01h 27mTutorial: Incentives for Collaborative Learning and Data Sharing, Part II
27mInterpretability and explainability in machine learning
35mTaking a peek under the hood - Interpreting black box model
27mFast, intuitive feature selection via regression on Shapley values
39mExplainable AI as a Conventional Data Analysis Tool
38mTowards Privacy-Respectful Cyber-Physical Systems: Two Perspectives
31mBeyond the Black Box: Interpreting ML Models with SHAP