Kafka Streams and KSQLDB

on YouTube ↗01h 07m

" summary

Kafka becomes an end-to-end streaming ETL: producers write to partitioned topics (default retention one week), consumers form group IDs and rebalance, and streaming is simply a consumer+producer that can filter, aggregate and write to new topics. State lives in a local RocksDB backed by a changelog topic so aggregates survive failover. Demos create topics with three or 25 partitions, run producers/consumers, and run KSQLDB on Avro data (pull vs push/emit changes persistent queries). Connectors persist results to S3, Snowflake, MongoDB.

AI-written summary · about

" chapters (12)

1. 0:00Introduction to Kafka
2. 1:40Kafka Streams overview
3. 3:47Kafka internals and partitions
4. 16:12Producers and consumers demo
5. 27:54Streaming fundamentals
6. 32:06State and compaction in streams
7. 41:23State stores and fault tolerance
8. 43:44Windowing in Kafka Streams
9. 53:24KSQLDB introduction
10. 59:25KSQLDB demo and queries
11. 1:04:02Architectural patterns with Kafka
12. 1:07:09Q&A and closing

" related talks

• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 42mdistributed stream processing for everyone with KSQL #bbuzzMichael Noll · Berlin Buzzwords 2018
• 41mKafka as a Platform: the Ecosystem from the Ground UpRobin Moffatt · ApacheCon@Home 2021
• 47mOne Does Not Simply Query a StreamViktor Gamov · Voxxed Days CERN 2025
• 41mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia November 2020