Large Scale Data Validation (with Spark and Dask)
summary
Compares Great Expectations (pandas + Spark) and Pandera (pandas only), shows tradeoffs: Great Expectations gives data docs and detailed failure output and suits weekly model training, Pandera is lightweight and fast for a 10‑minute inference pipeline. Demonstrates running Pandera on Spark via Fugue so pandas validators scale, and builds a pipeline that applies per-partition custom validations (different price ranges per state) in parallel on Spark — the approach works in the case study.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:11What Data Validation Is
- 3:57Common Validations
- 5:57Expectation Methods
- 7:46Expectation Config
- 8:26Data Profiler
- 9:46Pandaras
- 10:15Example Code Snippet
- 14:24Sample Code
- 20:37Validation by Partition
related talks
05mData Quality and Validation in ML Pipelines
36mScaling data workloads using the best of both worlds: pandas and Spark
47mEnforcing Data Quality in Data Processing and ML Pipelines with Flyte and Pandera
42mDeciphering the way data is tested
31mComparing the Different Ways to Scale Python and Pandas Code
28mData Validation for Feature pipelines: Using Great Expectations and Hopsworks
24mAn Introduction to PySpark
29mScaling Machine Learning Workflows to Big Data with Fugue