Stream Processor as a Database: Building Online Applications
summary
Claims that serving live aggregates directly from the stream processor removes the KV-store bottleneck: prototype uses Apache Flink + Kafka with a Queryable State extension (PR 3779) to answer queries from operator state instead of flushing millions of keys to Redis/Cassandra. Measured issues: external sinks topped out around 280k updates/sec for high cardinality (example 500M keys/hour), while Flink could push to ~15M events/sec; serving state in-process matches processor throughput. Uses async checkpoints/snapshots for durability; caveats include read-load contention, limited replication/read replicas, and no built-in auth yet.
AI-written summary · about
chapters (9)
- 0:00Introduction to stream processing
- 1:44Flink architecture basics
- 4:05Classic streaming architecture
- 10:06Performance and bottlenecks
- 15:53Queryable state paradigm
- 22:46Implementation details
- 26:05Why queryable state is efficient
- 34:05Key takeaways
- 35:56Q&A session
related talks
43mThe Stream Processor as a Database
42mStream Processing with Apache Flink
40mThe Seven Deadly Sins of Streaming
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
34mChasing the Stream Processing Utopia
21mStream Processing and SQL
39mEvaluating Streaming Framework Performance for a Large-Scale...
29mApache Kafka and the Next 700 Stream Processing Systems