Stream Processing, Going Beyond Pub/Sub with Kafka Streams and...

on YouTube ↗03h 00m

" summary

Concrete stream-processing pipelines using Kafka ecosystem components are presented: CDC with Debezium and Kafka Connect into Avro with Schema Registry, ingesting sources like a Pac‑Man game and Norwegian boat GPS into Kafka topics; processing with Kafka Streams and ksqlDB performing joins and windowing, using RocksDB state stores with standby replicas for recovery. The talk covers operational tradeoffs (ordering, late arrivals, windowing, exactly‑once), performance at LinkedIn scale and partitioning cautions, and shows real failure modes and fixes demonstrated on Confluent Cloud.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 18:24Streaming key concepts
3. 1:00:10Pac-Man demo with Kafka
4. 1:38:26Q&A
5. 1:49:19Aggregations and testing
6. 2:51:57Summary

" related talks

• 01h 29mEvent stream processing using Kafka StreamsFredrik Vraalsen · Berlin Buzzwords 2018
• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 01h 07mKafka Streams and KSQLDBDaniel Hinojosa · Great International Developer Summit 2024
• 50mApplication development and data in the emerging world ...Neha Narkhede · Berlin Buzzwords 2016
• 06mJay Kreps interviewJay Kreps · Data By the Bay 2016
• 55mDemystifying Stream Processing with Apache KafkaEwen Cheslack-Postava · Philly ETE 2016
• 45mApplications in the Emerging World of Stream ProcessingNeha Narkhede · GOTO Chicago 2016
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18