Use frameworks to test Apache Airflow data pipelines
summary
A tiny demo project builds a two-task math DAG with a custom BasicMath operator that pulls a random number from an API and writes to local Postgres, and it runs locally with Astro CLI and VS Code debugging via dag.test(). CI runs astrod def pytest covering DAG validation, unit tests and integration tests; a deliberately broken catchup=true test fails, while an astrod def upgrade test reports no DAG errors upgrading to runtime 10.3.
AI-written summary · about
chapters (17)
- 0:00intro
- 0:20preamble
- 0:31agenda
- 1:07what is airflow? who is astronomer?
- 9:33why test data pipelines?
- 10:53local development and testing
- 13:33astro cli and airflow cli testing features
- 16:23dag.test()
- 17:45ci/cd and testing
- 18:08from dev to staging
- 19:16dag validation testing
- 21:26unit tests with airflow
- 22:44integration tests with airflow
- 24:06the cf in ci/cd - our code on the way to the cloud
- 25:38code getting promoted to prod
- 26:26demo
- 45:49take home message
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
38mAirflow DAGs Made Easy: Introducing the Astro SDK
29mDagster: a Framework for Data Processing Applications
22mOrchestrate your data pipelines using Apache Airflow
01h 18mIntroduction to orchestration ETL processes with Apache Airflow
35mBecome a Conductor with Apache Airflow
45mDefining data pipelines
34mData Ingestion in Python using Apache Airflow