distributed stream processing for everyone with KSQL #bbuzz

on YouTube ↗42m

" summary

KSQL is a streaming SQL engine that turns what would be dozens of lines of Java or Scala into a single SQL statement that runs against Apache Kafka. It runs as a JVM server using Kafka Streams, keeps local state in RocksDB local state backed by Kafka changelog for fault tolerance and rebalances work across instances as consumer groups. Supported access is via CLI, REST, UI, supports JSON/Avro/CSV today, joins, aggregates, windowing, stream‑table joins, and headless or interactive deployment.

AI-written summary · about

" chapters (10)

1. 0:00Start
2. 3:56Motivating example
3. 6:57Stream processing with Kafka
4. 7:21Easier, faster workflow
5. 9:50KSQL for Data Enrichment Join data from a variety of sources to see the full picture
6. 11:08KSQL for Anomaly Detection Aggregate data to identify patterns or anomalies in real-time
7. 12:01KSQL for Data Transformation Quickly make derivations of existing data in Kafka
8. 28:21Example: CDC from DB via Kafka to Elas
9. 29:15Example: Real-time Data Enrichment
10. 38:21Fault-Tolerance, powered by Kafka A key challenge of distributed stream processing is fault-tolerant state Server

" related talks

• 01h 07mKafka Streams and KSQLDBDaniel Hinojosa · Great International Developer Summit 2024
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 47mOne Does Not Simply Query a StreamViktor Gamov · Voxxed Days CERN 2025
• 54mHow do you query a stream?Viktor Gamov · GeeCON 2024
• 45mApplications in the Emerging World of Stream ProcessingNeha Narkhede · GOTO Chicago 2016
• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 48mOne Does Not Simply Query a StreamViktor Gamov · JDD 2024