Engineering data quality
summary
Argues that data quality is an engineering problem with concrete tactics: use Luigi or Airflow for orchestration, emit counters and invariants to monitor assumptions, prefer a left join and repair logic over silent loss in inner joins, and run parallel fast and slow windows so financial reports wait while dashboards get quick estimates. Cites tooling like Spark, Kafka, Deequ and Great Expectations and the striking rule of thumb that about 10% of ideas are good, so measure and guard what downstream actually consumes.
AI-written summary · about
chapters (40)
- 0:00Intro
- 0:19Data value requires data quality
- 1:25Big data - a collaboration paradigm
- 2:49Data pipelines
- 4:28Data platform overview
- 5:46Data quality dimensions
- 6:54The truth is out there
- 7:24Truth mutated
- 8:22Not the whole truth
- 9:15Something but the truth
- 10:26Hearsay
- 12:16Events vs current state
- 13:30Monitoring timeliness, examples
- 14:30Ensuring timeliness
- 15:02Design for testability
- 15:43Potential test scopes
- 16:26Recommended scopes
- 16:49Scopes to avoid
- 17:26Testing single batch job
- 19:03Measuring correctness: counters
- 21:30Measuring correctness: pipelines
- 22:15Conditional consumption
- 23:24The unknown unknowns
- 24:02Fuzzy products
- 24:38Converting fuzzy to binary
- 25:30Golden scenario suite
- 26:07Weighted quality sum
- 26:45Testing with real world/production data
- 27:20Data completeness
- 27:42Incompleteness recovery
- 28:57Fast data, complete data
- 29:17Things to plan for early
- 30:16Does anyone care about data quality?
- 30:421999: Does anyone care about code quality?
- 31:26Code quality 1999
- 32:22Data quality 2019
- 33:06Changing culture bottom-up
- 34:28Repeating the success?
- 34:49Resources, credits
- 35:26Tech has massive impact on society
related talks
58mKEYNOTE: You Are Overthinking Data Quality
35mApplying Data Quality Monitoring
53mThe State of Data Engineering
28mEngineering Data Reliably Using SLO Theory
25mWhy Software Engineering Best Practices Fail in Data Engineering
41mData engineering without walls
41mData Quality: The Holy Grail For A Data Fluent Organization
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines