Streams Must Flow: Fault-tolerant streams processing apps on Kubernetes
summary
Shows a working Kafka Streams app (movie ratings) deployed to Kubernetes, using local RocksDB state stores and Kafka changelog topics so state stores replicated via changelog topics rather than external DBs. Claims parallelism equals topic partitions and tasks, so partitions = max parallel tasks; demonstrates scaling replicas until consumers hit partition limits and a real failure mode where pods stay Pending because of node/affinity constraints. Recommends StatefulSets and anti‑affinity, and toggling log.compaction/log.enable to speed restarts.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 3:24Thank you
- 5:03Github repository
- 6:46Kafka
- 10:53Streaming Pipeline
- 14:51Consumer Group
- 16:46Partitions
- 17:54Scaling
- 20:11Craftworld
- 21:40Recovery time
- 23:43Anti affinity rule
- 25:00Demo
- 32:30Conclusion
related talks
40mDeploying Kafka Streams Applications...
30mDeploying SQL Stream Processing in Kubernetes with Ease
41mWhat If We've Been Scaling Stream Processing Wrong All Along
40mKafka, DevOps...And Resilience for All
31mEvent driven communication with Kafka on Kubernetes
13mScaling Kafka on Kubernetes
41mMicroservices and Event Sourcing with Kafka and Kubernetes
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes