Keep it Clean: Why Bad Data Ruins Projects and How to Fix it
summary
Industrial projects lose enormous value to dirty data—management reports cite 3.1 trillion dollars lost and about 27% flawed records—while audits find 2.5–3% wrong names in COMPAS prison files and lifeboat IDs in Titanic data that flip survival labels. Examples include Tay.ai chaos, biased AWS hiring models, Babylon Health overstated 98% claims, ImageNet and Google Translate gender bias, and adversarial image failures. Practical fixes are concrete: visualize data, prefer simpler models, rescale features, handle nulls with imputation or SMOTE, detect anomalies, and expect ~60% time cleaning in real projects.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:24HOW BAD DATA AFFECTS RESULTS
- 18:23A GUARDIAN REPORTER FINDS...
- 24:13WHY BAD DATA AFFECTS RESULTS
- 28:24VISUALISING DATA
- 29:05SIMPLE MODELS
- 31:39DATA LEAKAGE
- 34:39FOONG MISSING DATA
- 36:55NOISE: WHAT IS NOISE?
- 40:58ANOMALIES (AKA OUTLIERS)
- 45:24NORMALITY
- 48:08CONCLUDING REMARKS
related talks
44mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
49mData Cleansing & Beyond: How to Address Data Debt for AI
31mThe Hidden Costs of Data Quality - Tackling Common Data Challenges in ML
25mWhy Good Models Fail After Deployment
35mBad data - anecdotes and examples from the real world
34mData Science beasts (failures) and where to find them
31mAvoid mistakes building AI products
23mHow to fail AI implementations