Testing Data Pipelines
summary
Covers concrete tactics: run end-to-end tests by snapshotting a few hundred–thousand rows, use snapshot testing to catch regressions across releases, add functional tests to execute small task groups, and write fast unit tests after separating orchestration from logic. Mentions tools: Airflow, Data diff, pandas assertions, soda core and great expectations, an in-house litmus and a dagline abstraction. Calls out scale — several terabytes per day — and a surprising Snowflake testing bill (~$20K), plus quotas and runner limits to control cost.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:28From scripts to team pipelines
- 2:52Testing challenges
- 3:56End-to-end testing
- 6:21Snapshot testing
- 8:07Functional testing
- 11:44Unit testing SQL
- 14:52Testing in production
- 17:19Data quality checks
- 19:30Testing framework
- 20:53Q&A
related talks
39mModern Data Pipelines Testing Techniques: A Visual Guide
20mBuilding a Data Pipeline with Testing in Mind
20mShift-left testing : Building reliable Data Pipelines
30mHow to Test BigData/ETL Solutions?
38mBuilding Data Pipelines in Python
39mWhat Can Go Wrong With Testing an ETL Pipeline?
38mBuilding Data Pipelines in Python
28mHow to not pull your hair out while providing data to the business