Netflix Keystone - Real-time Stream Processing Cloud Service
summary
Netflix Keystone is a multi-tenant, self-serve stream-processing service built to handle 1.3 trillion events/day (≈3 PB in, ≈9 PB fan-out) and keep four nines availability while losing <=0.1% during migrations. It runs Kafka fronting, routers and Flink workers on the Titus container runtime with Docker, Mesos-era schedulers and Spinnaker deployments, offers at-least-once semantics with checkpoints and savepoints, supports DSL, point‑and‑click and custom code, and automates failover and rolling upgrades — and it works in production.
AI-written summary · about
chapters (11)
- 0:00Start
- 2:57Keystone Pipeline
- 4:07Point-and-Click Pipelines
- 4:40Write Custom Code
- 5:08Point-and-Click Pipelines and the Dsl
- 7:16Filters
- 16:36Smart Deployments
- 30:07Deployment
- 36:33Metrics and Monitoring
- 37:02Self-Service Management Ui
- 41:16Smarter Deployments
related talks
19mKeynote: Making and Keeping Netflix Highly Available
31mTale of Stateful Stream to Stream Processing
27mBehind the Scenes: Running Critical Services @ Netflix
37mBuilding and operating large-scale streaming applications at King
48mCloud Native Event Streaming with Kafka & Open Standards
30mEvent Streaming and the Future of Applications
44mTechniques Netflix Uses to Weather Significant Demand Shifts
26mHow Streaming Helps Your Staging Environment and Sandboxes Stay Up To Date