How to build Data Lineage on Apache Spark logs
summary
Describes building a Data Lineage tool for Apache Spark used in a large national data warehouse. Covers the problem of tracing data flow from sources to consumers and explains a fast, low-cost implementation approach for Spark logs to produce full lineage information.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 4:58Problem and requirements
- 11:40Tech stack and architecture
- 13:11Spark logs collection
- 17:13Parsing physical plans
- 21:51Lineage coverage and validation
- 23:39Use cases and case studies
- 24:20Q&A
related talks
34mColumn-level lineage is coming to the rescue
30mCross-Platform Data Lineage with OpenLineage, A Foundational...
46mCross-Platform Data Lineage with OpenLineage
41mSpeeding Up Data Pipeline Construction in Apache Spark
27mData Lineage: A Key Building Block For A Successful Data Governance Strategy
32mTruly Open Lineage
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
15mArchitecting Scalable Data Lineage Graph for Privacy Compliance and Agentic Analysis