Data Lineage with Apache Airflow using Marquez

on YouTube ↗45m

" summary

Marquez is a metadata service that connects to Apache Airflow via a Marquez Airflow library which inspects tasks, parses SQL to auto-register sources/datasets/jobs and pushes run state and dataset versions to a Postgres core while replicating lineage to a Cayley graph DB and indexing to Elasticsearch. It runs on Kubernetes with Helm charts at WeWork, links runs to GitHub commits, understands Kafka schema registry versions, and enables evented data triggers to replace polling and simplify identifying affected downstream jobs for quick backfills.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 4:56Why metadata
3. 5:09Data quality
4. 7:07Selfservice
5. 10:14Marquez
6. 13:18Marquez Integration
7. 14:16Core Data Model
8. 15:54Design Benefits
9. 18:14Collecting Data
10. 21:39Deploying Marquez
11. 22:44Integrating Marquez with Airflow
12. 26:11Operator abstractions
13. 28:09Metadata clocking
14. 28:54Multiple Airflow instances
15. 30:39Deployment process
16. 31:24Data triggers
17. 34:07Demo
18. 37:44Roadmap

" related talks

• 35mAutomating Airflow Backfills with MarquezWilly Lulciuc · Open AI + Data Forum @ Open Source Summit North America 2022
• 46mCross-Platform Data Lineage with OpenLineageJulien Le Dem · Berlin Buzzwords 2022
• 30mCross-Platform Data Lineage with OpenLineage, A Foundational...Maciej Obuchowski · SREcon25 Europe/Middle East/Africa
• 41mObservability for data pipelines with OpenLineageJulien Le Dem · Berlin Buzzwords 2021
• 32mTruly Open LineageMandy Chessell · Open Source Summit Europe 2022: Open AI + Data Forum
• 24mMetadata driven framework for data discovery and data qualityXinwei Jiang, Akira Takihara Wang · DataEngBytes 2022
• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022