Cross-Platform Data Lineage with OpenLineage
summary
OpenLineage defines a lightweight event model that captures start/complete/fail events for each run, attaching extensible facets with JSON schemas (schema, stats, git sha, query profile) so tools can publish exact inputs/outputs and metrics. Marquez ingests those events, keeps a versioned model (job versions, dataset schemas) in Postgres, and the demo stitches Airflow DAG tasks, a Spark job (logical plan, row counts), dbt models and custom JSON events into one graph. Integrations cover Spark, Airflow, dbt, Snowflake, BigQuery, Iceberg, Kafka; scale targets are hundreds of thousands of runs/day with retention policies.
AI-written summary · about
chapters (20)
- 0:00Introduction to data lineage
- 1:46Why lineage matters across teams
- 3:30Introducing OpenLineage standard
- 8:06OpenLineage data model explained
- 14:14Use cases for lineage metadata
- 14:34Marquez implementation overview
- 17:23Integrations with Airflow Spark dbt
- 24:53Live demo setup and overview
- 26:17Airflow lineage in action
- 27:19Spark lineage instrumentation
- 28:46Custom integration example
- 31:17Marquez UI and navigation
- 32:21Joining the OpenLineage community
- 33:09Q&A session begins
- 33:31Scaling lineage for large datasets
- 35:38Best use cases for lineage
- 39:10Marquez in data governance
- 41:15Handling mutable datasets
- 44:02OpenLineage ecosystem adoption
- 46:09Closing remarks and next talk
related talks
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
41mObservability for data pipelines with OpenLineage
32mTruly Open Lineage
34mColumn-level lineage is coming to the rescue
45mData Lineage with Apache Airflow using Marquez
35mAutomating Airflow Backfills with Marquez
35mHow to build Data Lineage on Apache Spark logs
33mLet's Do Data Lineage in Kafka, Flink and Druid!