The Stream Processor as a Database
summary
Argues that a stream processor can act as the database by answering live key lookups from its local state instead of writing every event to a KV store; uses Flink + Kafka, demonstrates a prototype at Twitter, and exposes queryable state so queries hit the worker holding the windowed count. Claims per-stream load drops from ~1M events/sec to writes proportional to unique keys — ~400M unique tweets/hour → roughly 28,000 writes/sec to a backing DB — and demos a Flink 1.2 snapshot with a Griffino plugin.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:53Agenda
- 1:42Demo
- 3:34Time Series
- 3:57Different Approaches
- 9:34Kafka Offset
- 9:49Failure
- 15:38Complexity
- 16:40Beta Architecture
- 22:40Beta Architecture Pros Cons
- 23:40Omega Architecture Pros
- 26:03Omega Architecture Cons
- 27:10Omega Prime
- 27:40Flink Direct
- 28:13Flink Direct Demo
- 29:01Flink Job Demo
- 32:56Flink State Demo
related talks
41mStream Processor as a Database: Building Online Applications
42mStream Processing with Apache Flink
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
40mThe Seven Deadly Sins of Streaming
47mOne Does Not Simply Query a Stream
21mStream Processing and SQL
39mChangelog Stream Processing with Apache Flink