Distributed Real Time Stream Processing: Why and How
summary
Compares concrete systems — Spark Streaming (micro-batch), Flink (native streaming), Storm/Trident, Samza, Apex and Kafka — and shows how each implements state and failure recovery: upstream ack trees in Storm, micro-batch checkpoints in Spark, barriers/snapshots in Flink, Kafka offsets for recovery, and Apex markers. Reports latencies: micro-batching in seconds, native streaming down to milliseconds; enabling exactly-once semantics costs tens of percent (10–15% typical, Storm up to ~70%); example word-count shows stateful vs stateless slows Flink ~25% and Spark ~50%.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:38The Data Deluge
- 2:11Distributed Stream Processing
- 3:14Points of interest
- 5:17Runtime and Programming Model
- 6:02Native Streaming
- 6:59Micro-batching
- 8:23Apache Streaming Landscape
- 12:36System Comparison
- 17:15Fault Tolerance
- 22:02Managing State
- 25:44Counting Words Revisited
- 27:59Performance
- 30:49Project Maturity [Storm & Trident]
- 31:16Project Maturity [Samza]
- 31:41Project Maturity [Flink]
- 31:58Summary
- 34:53General Guidelines
- 35:29Recommendations [Storm & Trident]
- 36:12Recommendations [Spark Streaming]
- 36:53Recommendations Samza
- 37:46Recommendations [Apex]
- 38:27Recommendations [Flink]
- 39:13Dataflow and Apache Beam
- 41:43Questions
related talks
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
29mApache Kafka and the Next 700 Stream Processing Systems
01h 01mStream Data Processing for Fun and Profit
42mStream Processing with Apache Flink
39mEvaluating Streaming Framework Performance for a Large-Scale...
48mUnlocking Real-time Insights: Enhancing Your Databases With Stream Processing
33mKafka streams at scale
40mThe Seven Deadly Sins of Streaming