Large-Scale Data Graph: Scale and Optimize Privacy & Security in Offline Data Systems
summary
Meta builds a unified Data Graph that maps warehouses, stream processors and observability as vertices and flows, storing class/type, global and internal IDs, and freeform metadata; lineage comes from four signals — instrumentation, deep SQL parsing, pipeline annotations and log analysis — and feeds an iterative Discovery API with automated heuristics plus human curation. The system supports high‑precision output/input closure, subfield ABAC via an object attribute store and a unified permission service, and already drives privacy, access control and downstream recovery workflows.
AI-written summary · about
chapters (8)
- 0:00Data graph at Meta
- 2:55Data graph model & architecture
- 6:27Lineage signal collection
- 8:58Privacy use cases
- 10:43Privacy beyond closure
- 11:25Security & access control
- 13:56Fine-grained access control
- 16:03Key takeaways
related talks
15mArchitecting Scalable Data Lineage Graph for Privacy Compliance and Agentic Analysis
29mModern Data Stack: New Terminology Made Clear
26mData governance in streaming at scale
36mIntroducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
25mBuilding the World’s Largest Identity Graph without a Graph DB
56mGraph Analytic Systems
27mData Mesh for the Cognitive City
25mMoving Data From Graph to the Warehouse, A Configuration Driven Approach