Building the World’s Largest Identity Graph without a Graph DB
summary
A production identity graph replaces a separate graph DB by running joins on SingleStore backed by GCS/S3 object storage, using domain partitioning and ephemeral clusters to process per-customer slices; spilling to disk lets window functions and joins exceed memory. Outcome: a 50 billion × 48 billion join in seconds, handling ~100 PB on object store across ~6,000 pipelines and thousands of tables, cutting copies from ~30 to 1 and shrinking troubleshooting from days to minutes while lowering cores and cloud spend.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:18About LiveRamp
- 2:35The Problem
- 3:50Why are we solving it
- 4:40We are solving customer problems
- 5:41Im a crazy objective guy
- 6:15Finding the right solution
- 9:00How we solved it
- 16:37How we moved it
- 19:22Data Transformation
- 22:52Conclusion
related talks
46mScaling Identification Solutions Using Graph DB
40mKeeping identity graphs in sync with Apache Spark
18mLarge-Scale Data Graph: Scale and Optimize Privacy & Security in Offline Data Systems
41mReal time identity graph
50mBuilding a customer 360 graph application with Amazon Neptune
37mGraph Computing at Scale
27mData @scale Seattle - Yucheng Low
44mHow We Store and Analyze a Large Social Graph