Observability for data pipelines with OpenLineage
summary
OpenLineage defines a JSON schema and start/complete events with a unique run id to capture runtime metadata so systems expose job runs, input and output datasets, schemas and query plans; Marquez implements the spec and stores versions so a run that changes output schema or row count is linked to the exact upstream run that produced it. Integrations exist for Airflow, Spark, BigQuery, Snowflake, Redshift, Kafka, with dbt and Great Expectations in beta, eliminating N² integrations across tools.
AI-written summary · about
chapters (5)
- 0:00Intro
- 0:55Building a healthy data ecosystem
- 12:32Open Lineage scope
- 21:41Facet examples
- 33:16Join the conversation
related talks
46mCross-Platform Data Lineage with OpenLineage
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
32mTruly Open Lineage
34mColumn-level lineage is coming to the rescue
45mData Lineage with Apache Airflow using Marquez
19mObservability for Data Pipelines: Monitoring, Alerting & Tracing Lineage
35mAutomating Airflow Backfills with Marquez
29mA Practical Blueprint for Evolving Observability Schemas