Netflix Keystone - Streaming Data Pipeline @Scale in the Cloud
summary
Netflix Keystone moves over 700 billion events per day (peaked at a trillion) with at‑least‑once semantics, using fronting and consumer Kafka clusters (4000+ brokers, 24 clusters across three regions), Samza jobs in Docker for routing, S3 as the canonical sink, and a weekly failover drill via Kafka Kong backup clusters; the pipeline supports multi‑tenant routing, immutable payloads with 10‑byte metadata, 1.3 PB/day processed, and hit availability and data loss targets during live migration.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 2:22New pipeline launch and scale
- 4:08Data flow and Kafka architecture
- 7:24Kafka operations and failover
- 10:57Routing service and Samza setup
- 14:51Samza optimizations and patches
- 16:58Metrics dashboard and cost savings
- 18:47Future plans and Apache Beam
related talks
49mData Platform Architecture, Evolution, and Philosophy
42mHow Netflix Built a Real-Time Distributed Graph for Internet Scale
34mHow Netflix Delivers Key-Value and Time-Series Storage at Any Scale
34mChasing the Stream Processing Utopia
38mHow to handle 100k events per second
29mApache Kafka and the Next 700 Stream Processing Systems
57mLessons in Building a System that Processes More than 70 Billion Events Daily
46mCassandra Serving Netflix @ Scale