Structure and Interpretation of Stream Processing
summary
Compares concrete systems (Flink, Spark Streaming, Storm, Heron, Samza) and shows how choices change guarantees: Flink and Spark offer exactly‑once semantics, Storm/Heron default to at‑least‑once. Distinguishes execution models—micro‑batch (Spark) trades latency for throughput and easy fault recovery, event‑at‑a‑time (Flink/Storm) gives low latency but harder recovery. Details: managed state with RocksDB, checkpointing and Kafka-backed logs for replay, sliding windows (example: top Twitter hashtags every 10s over a 60s window), backpressure strategies and failure recovery tradeoffs.
AI-written summary · about
chapters (9)
- 0:00Introduction to stream processing
- 1:44Evolution of stream processors
- 8:09Programming abstractions overview
- 17:29Notion of time in streams
- 23:48Runtime and execution models
- 29:54Processing guarantees and fault tolerance
- 35:40Flow control mechanisms
- 39:19Trends and future directions
- 41:02Key takeaways and summary
related talks
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
38mEvery Event, Everywhere, All at Once
25mMixing data streams at scale
29mStreaming & Eventing for Top Management
01h 04mState or Events? Which Shall I Keep?
36mWhen ordering matters
37mKeynote | Flink First: Designing Modern Trading Systems
34mReactive Architecture