The final stage of grief (about bad data) is acceptance
summary
Practical recipe: write a probabilistic model of how reality becomes the recorded rows and infer its hidden parameters. Uses PyMC to infer a ~10% data loss, turning an observed 15% conversion into an estimated ~16.7% true rate; applies low-rank matrix completion on device-string collisions to merge noisy identifiers; fits survival curves to correct time-delay bias in early conversion rates; treats selection bias (Heckman problem) as unsolved for tree ensembles, not just a bug to be ignored.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:22A recipe for bad data
- 4:01Funnel analysis
- 5:19Where does the data come from?
- 8:12Model the data generating process
- 10:36Modeling the data
- 12:19Model your fundamental relationships
- 13:18Problem: Identify phishing and fraud
- 17:33Time for linear algebra
- 20:18Low rank approximation
- 23:54Collaborative filtering, simple version
- 27:52Pervasive problem in real world
- 29:27Concrete version of the problem
- 31:25Survival models
- 32:25Error correction
related talks
44mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
33mHidden Biases In Data: Moving Beyond Black Box Models
11mCausal Inference With Corrupted Data: Measurement Error, Missing Values, Discretization, And...
18mThe Importance of Modeling Data Collection
49mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
25mWhy Good Models Fail After Deployment
01h 24mBayesianism and Survival Analysis
39mDoing Data: The Critical Process of Data Preparation (...)