Help I need a stream processor - learning how to chose ...
summary
Deploys a smart‑meter example where commercial meters jump from one record per month to ~8,500/month and residential to ~2,500–3,000/month, then compares Apache Spark Streaming, Storm, Flink, and Samza on concrete behaviors: Spark is microbatch only and uses ingestion time, Flink supports event time, out‑of‑order data, snapshots and queriable state, Storm logs and drops late tuples and needs Trident for stronger guarantees, Samza lacks sliding/count windows and exactly‑once. Checkpointing, window types, and semantics determine which actually works in production.
AI-written summary · about
chapters (9)
- 0:00Introduction to stream processors
- 2:10Choosing a streaming framework
- 3:55Understanding time in streams
- 7:45Windowing types explained
- 12:01Handling out-of-order data
- 15:17Processing semantics guarantees
- 21:05State management in stream processing
- 25:53Checkpointing for recovery
- 29:09Making the final choice
related talks
45mData Stream Processing Concepts and Implementations
44mData Streaming: Which horse to bet on?
39mEvaluating Streaming Framework Performance for a Large-Scale...
28mSweet Streams Are Made Of These
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
40mRobust Stream Processing with Apache Flink
41mThe Engineer's Guide to Streaming
43mThe Stream Processor as a Database