Real time identity graph

on YouTube ↗41m

" summary

Adobe builds a real‑time identity graph on Flink that replaces a weekly 40 TB Spark batch job and runs at 50,000+ events/sec to serve 400M users across 2B+ devices. It combines a deterministic path (device ID + partner login hash) with an ML path (device/IP signals), uses a metadata working store and tumbling windows to dedupe spikes, relies on Flink backpressure and async sinks, runs on Kubernetes, and achieves near‑real‑time “personalization without a login” at production scale.

AI-written summary · about

" chapters (8)

1. 0:00Introduction to Adobe Experience Cloud
2. 5:22What is Adobe Experience Manager
3. 11:47Motivation behind identity graph
4. 15:01Why switch to real-time identity graph
5. 19:12Choosing Apache Flink for real-time
6. 21:19Real-time identity graph architecture
7. 28:36Lessons learned building real-time graph
8. 39:40Key takeaways and Q&A

" related talks

• 57mEmergence of Real-Time Analytics: Real-time Analysis of...S. Palthepu · Philly ETE 2016
• 18mReal-time Experiment Analytics at Pinterest with Apache FlinkParag Kesar, Ben Liu · Flink Forward Europe 2019
• 25mData: Moving with the flowAlex Walden · Big Data LDN 2022
• 45mApache Flink — Real-Time Data ProcessingGaël Renoux · Devoxx France 2024
• 41mBeyond the Hype: When Apache Flink Solves Real ProblemsNaci Simsek · Berlin Buzzwords 2026
• 43mLarge Scale User Behavior Analytics by FlinkHao Wu · Flink Forward Berlin 2017
• 08mReal-Time Data Systems: Speed, Scale, and ResilienceLakshmi Bhargavi Mullapudi · Conf42 JavaScript 2025
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019