Data Quality and Validation in ML Pipelines
summary
Compares three concrete tools and when they actually work: Great Expectations for pandas/SQL/Spark with human-written rules and HTML reports, practical for small–medium batches but not real-time or very large data; DQ (Amazon) running on Apache Spark, built for big AWS-scale datasets and tracking quality over time; TensorFlow Data Validation integrated with TensorFlow, auto-detecting schema drift and anomalies and best when the pipeline already uses TF. Lists specific failure modes: missing values, type flips, duplicates, outliers.
AI-written summary · about
chapters (8)
- 0:00Introduction and Importance of Data Quality
- 0:46Common Data Problems
- 1:20Introduction to Data Validation Tools
- 1:35Great Expectations Overview
- 2:01DQ Overview
- 2:28TensorFlow Data Validation Overview
- 2:52Tool Comparison and Recommendations
- 3:39Key Takeaways and Conclusion
related talks
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
26mLarge Scale Data Validation (with Spark and Dask)
01h 24mData Validation for Data Science
30mHow to Properly Test ML Models & Data
35mApplying Data Quality Monitoring
34mPanel 02 - Data Quality & Feature Stores
18mAutomate the boring ML stuff with pipelines
23mDDataflow: An open-source end to end testing from machine learning pipelines