Large-scale Data Quality Verification (...)
summary
Open-source library deequ lets teams unit-test your data by computing column-level metrics (completeness, uniqueness, histograms) on Apache Spark and evaluating user-defined constraints; it supports incremental state summaries so daily partitioned data avoid full rescans and a metrics repository enables simple anomaly checks (online mean/std or threshold) against historical values. Code examples show completeness and distinct-count assertions, histogram percentages, and a file-system backend; several teams already run it and report fewer runtime crashes and faster debugging.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:30What's in it for me?
- 1:22Why should we care about Data Quality?
- 4:28Quality Assurance
- 5:31Overview of Deequ
- 7:49A Unit Test for Data
- 10:21Data quality verification for partitioned data
- 13:20Continuous data quality verification
- 15:46Summary
related talks
39m"Unit tests for Data" with Deequ
34mPanel 02 - Data Quality & Feature Stores
42mDeciphering the way data is tested
15mQuick Guide on Data Quality Assurance
21mBuilding Trust in Your Data - Improving Your Data Quality with Automated Framework on AWS
28mHands-on Workshop: How To Perform Data Quality At Scale
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
53mData Mesh in Practice - Assuring Data Quality at Scale