Kafi Streams: Complex Stream Processing Made Simple
summary
Kafi Streams is an open source Python library built on PyDBSP that treats streams as weighted sets so streaming behaves like batch: strong, deterministic updates instead of eventual drifting results. It runs a word‑count with Z-sets and supports retraction (deletes encoded as negative weights). A contrived bank transfers test (100k messages) shows Flink produces fluctuating intermediate totals while the DBSP-based pipeline keeps the global sum zero and steady memory use (~170 KB); Kafka sources/sinks, snapshotting and GitHub examples exist, but time windows and spill‑to‑disk are still unfinished.
AI-written summary · about
chapters (11)
- 0:00Introduction to Kafka Streams
- 4:59Leaky abstractions in stream processing
- 9:31Enter database stream processing
- 11:15DBSP core concepts explained
- 17:57Introducing Coffee Streams
- 22:49Bank transfer example breakdown
- 25:16Flink demo and inconsistency
- 30:00Kafka Streams vs Flink results
- 33:09Memory efficiency with DBSP
- 36:18Current Coffee Streams features
- 38:23Q&A session