Preventing, Diagnosing & Curing Bad Data
summary
Explains how bad data enters each stage of the data lifecycle and gives concrete fixes: detect pipeline logging gaps by comparing source volumes, catch timezone and aggregation mismatches with cross‑source volume checks, and prevent recurring errors by shipping reusable transformation modules plus automated alerts and docs. Cites experience on subscription products and a product serving 400 million end-to-end users, shows failures like a pie chart not adding to 100 and broken dashboards, and reports that governance, ownership and regular audits actually stop repeat incidents.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 2:22Why bad data matters
- 11:13Data lifecycle and prevention
- 27:47Diagnosing data issues
- 34:13Curing and preventing data problems
- 40:58Q&A and closing
related talks
41mData Quality Management: The Good, The Bad, and The Messy
18mData quality: the key to long term happiness
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
30mDesigning Resilient Data Pipelines
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
38mEventually, time will kill your data pipeline
39mBREAKING YOUR DATA TEAM OUT OF THE SERVICE TRAP
27mData team strategies: unleashing your team