Garbage data in, garbage models out
summary
Builds two models for ad auctions: a classifier for whether an ad sells and a regressor for price, and shows concrete failures and fixes — mislabeled region codes, a feature that’s populated only after sale causing data leakage, duplicate auction IDs leaking into test sets, and a currency-conversion bug creating outliers. Recommends k-fold cross-validation and an orbit-X ablation to find useful features, collapses 40 messy app categories into ~10 via feature engineering, handles an 82% imbalance with over/undersampling, and notes tree models beat deep nets on tabular data.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:49Learning machine learning
- 2:31Basic model types
- 6:10The danger of assumptions
- 7:11What do our features mean?
- 12:12How do we see how good a model is?
- 13:20Reserve data for testing
- 21:59Missing values
- 25:50Detecting outliers
- 27:58Adjusting for class imbalance
- 30:00Curse of dimensionality
- 31:22Feature engineering
- 32:33Creating generalisable models
- 34:55Selection bias
- 36:24Survivorship bias
- 38:36How does your model make predictions?
- 40:42Are you using the right metrics?
- 43:34Is newer and shiner better?
related talks
18mBuilding Models Using Ad Sales as an Example
45m"Breaking black-box machine learning"
33mMachine Learning to moderate ads
35mWhat Failure Taught Me About Building High-Stakes Models
37mOptimal on Paper, Broken in Reality
48mThe Missing Step in Moving from Data to Models
47mBreaking Black-Box AI
52mBreaking black-box AI