Stream Processing Smackdown: Kafka Streams vs. Apache Flink

on YouTube ↗51m

" summary

Compares Kafka Streams (a JVM library tightly coupled to Kafka) and Apache Flink (a standalone cluster engine) by showing concrete choices: Kafka Streams uses local RocksDB state with changelog topics for backup and built‑in exactly‑once via Kafka transactions, Flink uses checkpoints and watermarks with task managers for flexible connectors and advanced windowing. Demonstrates a delayed‑flights example (15‑minute threshold), shows state materialization, repartition topics, and a fat JAR submitted to Flink; both approaches work but recovery and scaling tradeoffs differ.

AI-written summary · about

" chapters (15)

1. 0:00Stream processing battle intro
2. 1:46What is stream processing
3. 2:48Kafka deep dive
4. 8:53Kafka producers consumers partitions
5. 15:25Contenders: Kafka Streams vs Flink
6. 17:31Kafka Streams architecture
7. 22:56Flink architecture
8. 26:50State management comparison
9. 30:23Scalability comparison
10. 33:57Window operations
11. 36:46Exactly once guarantees
12. 40:03Kafka Streams vs Flink choice
13. 41:51Flink vs Kafka Streams choice
14. 43:16Live demo
15. 48:13Q&A

" related talks

• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 42mDistributed Real Time Stream Processing: Why and HowPetr Zapletal · Scala Days Berlin 2016
• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015
• 39mEvaluating Streaming Framework Performance for a Large-Scale...Ron Crocker · Flink Forward Berlin 2016
• 41mStream Processor as a Database: Building Online ApplicationsStephan Ewen · Berlin Buzzwords 2016
• 31mEvent Streaming with Kafka StreamsViktor Gamov · Scale By the Bay 2020
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Berlin Buzzwords 2021
• 43mUnified stream and batch processing with Apache Flink’s relational APIsFabian Hueske · Flink Forward Berlin 2017