How Do You Query a Stream?
summary
Kafka topics become queryable in four different ways with clear tradeoffs: Kafka Connect → Postgres copies events into a relational DB for familiar key lookups but hits scale limits; KTable + interactive queries (Kafka Streams) builds in-memory, sharded key stores that serve fast lookups but require a JVM topology per query pattern; streaming SQLs like Materialize materialized views and Delta/RisingWave provide persistent, SQL-defined views for both push and pull queries; real-time engines such as Pinot: 10 ms on PB target high-concurrency, low-latency analytics for user-facing features.
AI-written summary · about
chapters (17)
- 0:00Introduction to stream querying
- 1:46Evolution of data infrastructure
- 10:06OLTP vs OLAP on streams
- 13:59Querying streams options overview
- 14:20Kafka Connect with relational DB
- 18:12Kafka Streams for transactional queries
- 22:01Kafka Streams for analytical queries
- 25:16Streaming SQL solutions comparison
- 28:03Materialize as streaming warehouse
- 30:10Delta Stream cloud-native SQL
- 32:18Rising Wave distributed SQL
- 33:45KSQL DB confluent streaming SQL
- 35:51Cloud data warehouses for streams
- 40:28Data lakes and Iceberg/Hudi/Delta
- 44:05Real-time analytics databases
- 48:22Technology selection guide
- 50:53Closing and Q&A
related talks
08mPREVIEW: Kafka Stream Processing for Everyone with KSQL
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
39mKafka as a Platform: The Ecosystem from the Ground Up
31mEvent Streaming with Kafka Streams
21mStream Processing and SQL
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
43mThe Stream Processor as a Database