"Unit tests for Data" with Deequ
summary
Demonstrates Deequ running on Apache Spark with a declarative API plus small user lambdas to enforce concrete constraints—completeness, uniqueness, ranges—and sketch‑based statistics for fast approximations. Shows a Databricks JSON metric repository that stores historical metrics, and anomaly checks (10% threshold) that compare runs; when IDs go null or city values change the verification fails and the inference pipeline is blocked. Core is Scala with a Python binding (pydeequ) and it works end to end.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:48Why data quality matters
- 3:56Data as a living entity
- 9:13Data quality dimensions
- 12:26Deequ inspiration
- 17:26Deequ components workflow
- 22:44Live demo setup
- 26:03Running verification
- 28:14Analyzing test failures
- 30:08Constraint suggestions
- 31:14Q&A: Deequ vs others
- 33:23Q&A: Cloud & Spark
- 34:08Q&A: Failed pipelines
- 35:56Q&A: Corrupted data
- 37:48Q&A: Batch failures
related talks
21mLarge-scale Data Quality Verification (...)
34mTesting Big Data Applications (Spark, Dask, and Ray)
39mTesting Big Data Applications (Spark, Dask, and Ray)
21mData Observability with HEDDA.IO
23mDDataflow: An open-source end to end testing from machine learning pipelines
46mSimplifying Testing of Spark Applications
18mSimple and Effective Spark Tests with Spark-Test
39mUnit Testing for Data Scientists