Kafka streams at scale

on YouTube ↗33m

" summary

Explains using Kafka Streams for stateful real-time processing of multi-million event workloads, including an overview of event flow and Kafka Streams basics. Details practical challenges—fault recovery, horizontal scalability, cloud readiness, limitations of embedded RocksDB, and operating very large clusters—and describes code forks and engineering solutions implemented over the past year to address each issue, concluding with future work.

AI-written summary · about

" chapters (16)

1. 0:00Kafka Streams at Scale
2. 0:24Stateful processing of multi- million events in real time
3. 5:35Kafka Streams Challenges
4. 6:12Recovery
5. 8:07Cold Bootstrap
6. 9:38Horizontal Scalability
7. 10:13Repartitioning Logic
8. 10:53Dynamic Repartitioning
9. 12:09Scaling Lookups
10. 14:19Cloud Readiness
11. 14:35Rack/AZ Aware Task Assignment
12. 16:02Partial Partition Assignment Revocation
13. 20:04Large Clusters
14. 20:30Rebalance Time Reduction
15. 22:09Configs
16. 28:13Up Next

" related talks

• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015
• 01h 00mKafka as a Platform: the Ecosystem from the Ground UpRobin Moffatt · NDC Oslo 2020
• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 17mOvercome By Events - Apache Kafka And Event-oriented ArchitectureJay Kreps · Big Data LDN 2018
• 31mEvent Streaming with Kafka StreamsViktor Gamov · Scale By the Bay 2020
• 34mChasing the Stream Processing UtopiaKartik Paramasivam · Strange Loop 2018
• 41mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia November 2020
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020