Running a Massively Parallel Self-Serve Distributed Data System at Scale
summary
Keystone is Netflix’s cloud-native streaming pipeline that actually processes ~1.3 trillion events/day (over 500 billion generated daily) using Kafka + Flink + Titus, with ~4,000 Kafka instances and ~9,000 processing containers. It guarantees at least once delivery, uses checkpoint barriers, fan‑out to secondary clusters, declarative reconciliation and immutable container deploys for self‑healing and multi‑tenant isolation, and handles backpressure and cluster failures by routing to new clusters and replaying offsets.
AI-written summary · about
chapters (12)
- 0:00Start
- 6:17Data Driven Culture
- 7:07What Exactly Is Key to Keystone Streaming System
- 9:18Challenges and Solutions
- 16:14Delivery and Processing Semantics
- 17:30Event Loop
- 20:22Per Stream Monitoring and Alerting
- 26:56Total Infrastructure Scale
- 30:55Designing a Large-Scale Distributed System
- 33:02Infrastructure Upgrades
- 33:46Failure Modes
- 38:26Failure Cases
related talks
05mPREVIEW: Multi-Tenant, Multi-Cluster, Hierarchical Messaging
45mDeep dive into Unbounded Data Processing Systems
46mCassandra Serving Netflix @ Scale
50mFast Delivery
30mEvent Streaming and the Future of Applications
57mLessons in Building a System that Processes More than 70 Billion Events Daily
28mTransforming the Financial Services Industry with Event Stream Data
33mMicroservices, Events, and Breaking the Data Monolith with Kafka