Data Validation at Scale: Managing Data Quality in Complex Data Pipelines
summary
Built a monitoring/data science pipeline for web performance tracking that enforces constraints (Python schema checks) and runs continuous incremental validation rather than batch rechecks. Real results: half a billion data errors in one month, two thirds are aggregation stalls and many stem from missing unload beacons with 80–90% reliability; one customer spike jumps errors from ~100k to ~500k/hour and can stall the whole system. Practical rules include LCP must be 0–600000 ms and normalizing formats; the system scales by incremental checks, customer-sharded load shedding, and automated constraint inference for 150+ tracking attributes.
AI-written summary · about
chapters (7)
- 0:00Data validation at scale
- 2:14Why validation matters
- 8:53Four dimensions of quality
- 16:39Enforcing constraints
- 22:36Validating at scale challenges
- 34:11Key takeaways
- 36:58Q&A
related talks
45mValidating Big Data Jobs - Stopping Failures before Production
35mApplying Data Quality Monitoring
31mData Reliability Engineering: Secret Of Our Success
05mData Quality and Validation in ML Pipelines
18mData quality: the key to long term happiness
01h 24mData Validation for Data Science
23mChallenges of database testing
41mData Quality: The Holy Grail For A Data Fluent Organization