Measuring Model Fairness
summary
Introduces concrete fairness metrics—disparate impact (statistical parity), equal opportunity—and compares their tradeoffs with examples such as COMPAS error rates (false positives ~24% white vs ~45% Black, false negatives ~48% white vs ~28% Black). Demonstrates on anonymized Civis client data how label bias and sampling bias distort those metrics, shows when AUC and accuracy can mask disparities, and reports a case study that injects known biases to reveal which measures detect them.
AI-written summary · about
chapters (14)
- 0:00Model fairness measurement
- 1:05Why fairness matters
- 2:07Compass recidivism controversy
- 5:56Fairness metrics limits
- 6:40Ground truth rate differences
- 8:24Training data label bias
- 11:35Sampling process bias
- 13:25Model error consequences
- 14:49Human judgment in fairness
- 15:52Case study setup
- 19:48Testing fairness metrics
- 21:11Metrics fail on imbalance
- 22:36Fair model development
- 25:25Q&A
related talks
31mMeasuring Model Fairness
50mThe Measure and Mismeasure of Fairness
46mThe Measure and Mismeasure of Fairness
46mA look at the methods to detect and try to remove bias in ML models
40mQuestions and Practices to Make Algorithmic Decision Making More Fair
21mFairness and bias in AI
23mRemoving Unfair Bias in the Machine Learning Pipeline
42mFairness in Machine Learning: From Theory to Practice