Architecting Scalable Data Lineage Graph for Privacy Compliance and Agentic Analysis
summary
Meta builds an in‑memory lineage graph that ingests from 15+ lineage providers and processes billions of edges daily across web, warehouse, and AI systems. The system gets edges to fit in cache using integer compression, string interning, edge deduplication and selective materialization, then serves a prebuilt, validated graph with BFS traversal plus decoding, rule‑based pruning, fan‑out and cycle detection. Daily refreshes, a human feedback loop and SLAs yield sub‑second traversals that support interactive agentic analysis and reduce false positives.
AI-written summary · about
chapters (11)
- 0:00Data lineage graphs intro
- 1:16Tracking data flow importance
- 2:19Unifying lineage providers challenges
- 3:43Compressing billions of edges
- 6:30Stitching system layers data
- 8:15Accuracy in data flows
- 9:40Scaling versus traversal speed
- 10:44In-memory graph processing
- 12:30Fast compressed graph traversal
- 13:54Operationalizing graph at scale
- 14:57Prioritize quality and freshness
related talks
18mLarge-Scale Data Graph: Scale and Optimize Privacy & Security in Offline Data Systems
27mData Lineage: A Key Building Block For A Successful Data Governance Strategy
46mCross-Platform Data Lineage with OpenLineage
32mTruly Open Lineage
10mData Access Automation at Scale
28mBuilding A Thriving Data Culture That Ignites The Fire To Fuel Innovation
12mUsing GenAI to Accelerate Privacy Implementations
34mColumn-level lineage is coming to the rescue