Building the World’s Largest Identity Graph
summary
LiveRamp uses SingleStore plus object store (GCS/S3) to build an identity graph that operates at 100 petabytes of data by domain partitioning and ephemeral compute clusters. A proof shows a 50B x 48B join in seconds by sharding files across nodes and spilling joins to disk, replacing a Spark file pipeline of from 30 file copies to 1 and cutting a 10k‑core job into much smaller clusters. The pipeline runs ~6,000 pipelines, speeds troubleshooting from days to minutes, and reduces cost and carbon.
AI-written summary · about
chapters (10)
- 0:00Introduction to LiveRamp
- 3:01Why we needed change
- 5:49Choosing the right database
- 9:19SingleStore for batch processing
- 12:31Domain partitioning approach
- 16:46Scaling the identity graph
- 18:36Carbon footprint reduction
- 20:23Simplifying data pipelines
- 22:12Troubleshooting efficiency
- 23:17Results and future plans
related talks
41mReal time identity graph
40mKeeping identity graphs in sync with Apache Spark
46mScaling Identification Solutions Using Graph DB
27mData @scale Seattle - Yucheng Low
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
18mLarge-Scale Data Graph: Scale and Optimize Privacy & Security in Offline Data Systems
30mLarge Scale Graph Solutions: Use-cases and Lessons Learnt
50mBuilding a customer 360 graph application with Amazon Neptune