Evaluating Streaming Framework Performance for a Large-Scale...
summary
Three implementations run against the same Kafka stream: a baseline Java consumer, Apache Flink, and Apache Spark Streaming. Spark produces incorrect aggregations under load and is abandoned despite throughput; the baseline hits CPU-bound Kafka lag earlier. Flink correctly aggregates and sustains much higher rates — the experiment shows roughly 5x headroom before compute limits and then a network bottleneck at ~6000% scaled load. An AWS clone and a combined minute+hour topology are built and function.
AI-written summary · about
chapters (6)
- 0:00Introduction and challenges
- 4:03Data processing architecture
- 12:26Experiment design
- 23:00Framework evaluation
- 28:04Results and findings
- 32:42Q&A session
related talks
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
27mNavigating Streaming Infrastructure
44mData Streaming: Which horse to bet on?
29mApache Kafka and the Next 700 Stream Processing Systems
31mHelp I need a stream processor - learning how to chose ...
28mSweet Streams Are Made Of These
42mDistributed Real Time Stream Processing: Why and How