Data Validation for Data Science
summary
Tutorial shows why data validation prevents trivial failures and how to do it: use Great Expectations to build an expectation suite and run checkpoints that report failing rows (duplicate ids, unexpected categories); use Pandera schemas in training to enforce types, ranges, non-null and to catch bad rows for isolation; use Pydantic models for serving with Optional and enum fields, custom validators, and a coercion/strict strategy to convert or reject malformed inputs.
AI-written summary · about
chapters (39)
- 0:00Start
- 2:31Agenda
- 4:09Validation in Databases
- 7:23What Does Data Validation Actually Mean
- 8:27How Can Data Become Invalid
- 8:37Human Error
- 9:17Outdated or Expired Data Sets
- 10:45Reasons To Put Data Validation Steps in Place
- 12:12Data Drift
- 13:23Familiarity
- 14:09What Point in the Model Life Cycle Should We Implement Data Validation
- 15:19Model Training
- 16:08Hands-on Tutorial
- 17:01Install the Required Packages
- 19:43Create a Data Context
- 21:22Create a Data Source
- 22:00What Is a Data Source
- 22:34Define the Execution Engine
- 24:03The Data Connector
- 30:42Expect Column Values To Be Unique
- 40:02Solutions
- 41:28Validating the Data in a Data Checkpoint
- 47:05Pandira
- 48:33Train a Basic Model
- 49:48Pandera
- 50:44Create Pandira Schema
- 52:41Bad Data Schema
- 56:03Validate Acceptable Categorical Variables
- 56:58Coerce
- 59:57Hypothesis Testing
- 1:01:56Practice Defining Data Frame Schemas Using Pandira
- 1:07:20Solutions to the Exercise
- 1:07:53Checking for Range
- 1:10:09Model Serving
- 1:10:19Training Pipeline
- 1:16:09Custom Based Models
- 1:19:11Custom Validation
- 1:22:49Profiling Tools
- 1:23:16How Big Is the Community
related talks
05mData Quality and Validation in ML Pipelines
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
41mA Cookbook for Testing Data
35mApplying Data Quality Monitoring
58mKEYNOTE: You Are Overthinking Data Quality
35mAvoiding Bad Database Surprises: Simulation and Scalability
41mData Quality: The Holy Grail For A Data Fluent Organization
37mEngineering data quality