Making Data Quality an integral part of developing Machine Learning and Data Products
summary
Describes a concrete pipeline for California housing: a type-safe orchestration platform that wires typed Python tasks into reproducible workflows, plus a data-frame schema tester that asserts value constraints and dataset-level statistics. Claims that Flyte enforces typed inputs/outputs, enables task caching to skip recomputation, and preserves lineage and rerun recovery; that Pandera enforces per-column constraints and distributional checks. Demonstration shows the tools composing to catch bad values and speed repeated runs.
AI-written summary · about
chapters (7)
- 0:00Data quality in ML
- 2:12California Housing setup
- 3:36Production ML complexity
- 4:40Type safety in ML
- 7:49Flight orchestration tool
- 10:56Pandera data constraints
- 13:04Key takeaways
related talks
18mData quality: the key to long term happiness
22mProtecting data scientists in healthcare with type safety
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
44mData Quality Management
01h 02mShift Data Quality Left with Data Mesh Principles
26mUnlocking the Power of Data-Centric AI.
24mDataOps as a Service
27mLive in the Data Wild West: The Data Contracts Sheriff