Data Quality with Great Expectations and Airflow in a Reverse-ETL World
summary
Shows how Great Expectations plus Airflow gate reverse ETL pipelines into tools like Census after dbt models in Snowflake, preventing paid-ads from shipping only out-of-stock items (an incident that can cut marketing-driven revenue by up to 30–50%). Implements logic and business-metric checks (column types, in_stock flags, percent scaling), blocks downstream syncs and triggers Slack/email alerts on failures, and runs dry-run validation in a cloned staging warehouse and Airflow instance before merging to master.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:31Intro & Outline
- 1:40The Modern Data Stack
- 4:05The Power of Reverse-ETL
- 7:46Production Systems Are Tested
- 10:52Implementing Data Quality Tests
- 15:32Tests Serve as a Gateway
- 17:49Dry Runs in a Staging Environment
related talks
30mData Engineering building your BI infrastructure from scratch
52mBetter ETL with Managed Airflow in ADF
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
25mTesting Data Pipelines
23mData Team as an Optimisation Problem
30mHow to Test BigData/ETL Solutions?
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter