How Netflix Built a Real-Time Distributed Graph for Internet Scale
summary
Netflix builds a real‑time distributed graph (RDG) using Apache Kafka and Flink for near‑real‑time ingestion, a Key‑Value Abstraction Layer (KV doll) backed by Apache Cassandra with adjacency‑list edges, and Apache Iceberg for offline copies. The system holds over 8 billion nodes and 150 billion edges, sustains about 2M reads/sec and 6M writes/sec, spans ~27 namespaces on ~12 Cassandra clusters (~2,400 EC2 instances), uses per‑source Flink jobs plus Spark backfills, and operates at internet scale with tunable latency and TTL purges.
AI-written summary · about
chapters (8)
- 0:00Netflix and RDG intro
- 1:06Why Netflix built RDG
- 7:11RDG ingestion and processing
- 11:03RDG data model and storage
- 19:28Serving RDG data
- 20:52RDG scale and performance
- 22:15RDG challenges
- 39:02RDG effort and future
related talks
34mHow Netflix Delivers Key-Value and Time-Series Storage at Any Scale
20mNetflix Keystone - Streaming Data Pipeline @Scale in the Cloud
44mTechniques Netflix Uses to Weather Significant Demand Shifts
46mCassandra Serving Netflix @ Scale
19mStreaming Event-Time Partitioning With Apache Flink and Apache Iceberg
27mNetworks! project - real-time analytics that controls 50% of mobile network in Poland
24mHow Netflix enables
49mData Platform Architecture, Evolution, and Philosophy