Event stream processing using Kafka Streams

on YouTube ↗01h 29m

" summary

Hands-on workshop showing how Kafka Streams becomes the core of a production pipeline that processes 800 million events per day from collectors through Amazon Kinesis into Kafka, then routes, filters, enriches and aggregates with Kafka Streams with Java 8/Scala code. Examples include rekeying and repartitioning for joins, materialized state in RocksDB changelogs, windowed aggregations and KTables for per-site and per-hour counts, and practical notes on scaling, exactly-once semantics, and exporting results to S3 or downstream analytics.

AI-written summary · about

" chapters (9)

1. 0:00Introduction and setup
2. 11:30Why Kafka and Kafka Streams
3. 20:48Kafka Streams basics and code
4. 33:37Exercises and hands-on
5. 46:35Slides and resources
6. 55:06Aggregations and time windows
7. 1:09:48State stores and queries
8. 1:16:13Joining streams and enrichment
9. 1:26:09Q&A and closing

" related talks

• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 55mDemystifying Stream Processing with Apache KafkaEwen Cheslack-Postava · Philly ETE 2016
• 27mStream processing with Kafka streamsHamza Belmellouki · BlaBlaConf 2022
• 40mEvent Streaming Applications with Kafka Streams, Spring Kafka and ActuatorKo Turk · Voxxed Days Bucharest 2022
• 55mReal-Time Data Analysis with Apache KafkaRadosław Szmit · Confitura 2019
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 37mIntroducing Kafka Streams, new stream processing library ...Michael Noll · Berlin Buzzwords 2016