How do you query a stream?
summary
Argues there is no single right way — choices depend on trade-offs. Lists concrete options: Kafka Connect ships topics into Postgres or other OLTP stores (works well for gigabytes–hundreds of GB; terabytes hit the destination DB), Kafka Streams materializes key/value state stores and exposes interactive queries but requires deploying app code, streaming‑SQL systems (Materialize, Delta, RisingWave, ksqlDB) provide serverless or persistent materialized views via Postgres protocol, and real‑time analytics DBs (Apache Pinot, ClickHouse) give low‑latency indexed queries for high QPS.
AI-written summary · about
chapters (10)
- 0:00Introduction to stream processing
- 3:00From monoliths to streaming data
- 8:46Streams vs traditional databases
- 11:15Kafka Connect for data pipelines
- 16:32Kafka Streams for transformations
- 25:01Streaming SQL databases
- 30:40Cloud data warehouses and lakes
- 40:11Real-time analytics databases
- 48:35Choosing the right tool
- 51:47Q&A and closing remarks
related talks
47mOne Does Not Simply Query a Stream
48mOne Does Not Simply Query a Stream
01h 07mKafka Streams and KSQLDB
42mdistributed stream processing for everyone with KSQL #bbuzz
01h 00mIn Depth Kafka Streams
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
45mApplications in the Emerging World of Stream Processing
27mNavigating Streaming Infrastructure