Data quality: the key to long term happiness
summary
Argues that manufacturing-style statistical process control maps to data pipelines by versioning collection, processing and models, capturing anomalies instead of discarding them, and automating checks with json schema validation (example NPS 1–10 and ISO timestamp), profiling with Capital One profiler and Great Expectations, and lineage tracking via OpenLineage. Recommends supervised, semi-supervised or unsupervised tests, plotting SLOs and failure rates, quarantining and replaying bad records, and alerting conservatively to avoid noisy false positives.
AI-written summary · about
chapters (8)
- 0:00Introduction to data quality
- 2:05Lessons from manufacturing
- 5:12Process capability studies
- 6:56Data validation techniques
- 10:47Profiling and expectations
- 11:08Plotting and interpreting data
- 13:14Corrective actions for issues
- 15:00Key takeaways summary
related talks
41mData Quality Management: The Good, The Bad, and The Messy
20mScale-up your job satisfaction, not your software
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
44mData Quality Management
39mDoing Data: The Critical Process of Data Preparation (...)
44mPreventing, Diagnosing & Curing Bad Data
31mData Reliability Engineering: Secret Of Our Success
15mQuick Guide on Data Quality Assurance