Enforcing Data Quality in Data Processing and ML Pipelines with Flyte and Pandera
summary
Combines Flyte for type-checked workflows and Pandera for statistical schema checks to stop silent data errors; demo trains an sklearn RandomForest on the UCI heart disease CSV, surfaces parse errors (strings like "0.0" and "?" breaking int casts) and returns 82% test accuracy for the assembled pipeline. Shows Pandera custom validators, regex column rules, integration with Hypothesis for property-based unit tests, and notes Pandera currently operates on pandas (scaling to dask/modin/koalas is roadmap).
AI-written summary · about
chapters (15)
- 0:00Start
- 1:17Outline
- 4:24The Problem with Dataframes
- 9:53Compose Workflows: Connect Tasks Together
- 11:34Flyte's Capabilities
- 12:57High Quality Clean Data is Critical for Data Analysis and Modeling
- 13:56Flyte is a strongly-typed data pipeline framework
- 16:58Statistical Typing Specifying the properties of collections of data points
- 20:21Pandera in Practice
- 22:57Extend pandera's built-in validation checks
- 24:52Get informative errors if something goes wrong
- 26:50Test your pipeline code with schema hypothesis strategies
- 31:17Train a Model with Sklearn
- 31:54Architecture
- 46:18Questions
related talks
26mLarge Scale Data Validation (with Spark and Dask)
22mStatistical Typing: A Runtime TypingSystem for Data Science&Machine Learning
05mData Quality and Validation in ML Pipelines
01h 24mRobust and End-to-End Cloud Native Machine Learning & Data Processing Platform
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
18mAutomate the boring ML stuff with pipelines
39mFlyte: Production-Grade Data and Machine Learning Orchestration
01h 30mAdvanced Software Testing for Data Scientists