One Does Not Simply Query a Stream
summary
Compares concrete tools and tradeoffs for querying event logs: Kafka is storage not a database, Kafka Connect ships streams into existing stores so relational queries work but transformations are limited, Kafka Streams gives interactive queries and KTables but requires running custom Java topologies and only looks up by key, streaming SQL engines (Flink SQL, Materialize, RisingWave, DeltaStream) create materialized views for streaming SQL so you can run ad hoc aggregations, and real‑time OLAP systems like Apache Pinot provide native indexes and low‑latency analytical pulls.
AI-written summary · about
chapters (19)
- 0:00Introduction to querying streams
- 3:11From monoliths to streaming data
- 7:24Log-based systems vs messaging
- 10:39Challenges of querying logs
- 11:41Transactional vs analytical queries
- 12:47Kafka Connect for data integration
- 17:19Kafka Streams for processing
- 26:22Streaming SQL databases overview
- 27:52Apache Flink and Flink SQL
- 30:42Materialize for real-time views
- 33:35Delta Stream and cloud-native SQL
- 34:20Rising Wave for JVM alternatives
- 37:57KSQL DB for Kafka streams
- 40:08Cloud data warehouses
- 42:16Data lakes and ELT
- 43:21Real-time analytical databases
- 45:08Trade-offs and choosing solutions
- 46:52Community and support importance
- 47:56Closing and exploration tips
related talks
47mOne Does Not Simply Query a Stream
54mHow do you query a stream?
27mNavigating Streaming Infrastructure
01h 07mKafka Streams and KSQLDB
42mdistributed stream processing for everyone with KSQL #bbuzz
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
39mData Streaming? I don't even know her
01h 00mIn Depth Kafka Streams