How We Built SQL Rollups on Streaming Data
summary
Rockset implements SQL rollups on Kafka and Kinesis by running a decomposed SQL rollup query at ingest and finalizing aggregates at read time, cutting storage and speeding queries. An added aggregator service runs the ingest query, emitting documents keyed by a 128‑bit hash of grouping dimensions as _id so leaves can do read‑modify‑write updates. Partial aggregate states are stored (sum/count for avg), the search index omits nonfinal partials, and exactly-once per-shard sequence enforcement prevents double counting while handling out‑of‑order arrivals.
AI-written summary · about
chapters (9)
- 0:00Introduction to SQL rollups
- 2:26Why build SQL rollups
- 6:37Design principles for rollups
- 9:46Decomposing rollup SQL queries
- 12:36Executing SQL at ingest time
- 15:27Handling out-of-order arrivals
- 18:19Exactly-once write semantics
- 20:25Implementation summary
- 21:48Q&A session
related talks
39mHow we isolate streaming ingest from search using RocksDB
01h 07mKafka Streams and KSQLDB
31mBe a Productive Dev with Real time Analytics as an API
20mEnabling Machine Learning through Real-Time Data Processing
19mGetting from Schemaless Ingest to Fast SQL
42mdistributed stream processing for everyone with KSQL #bbuzz
47mOne Does Not Simply Query a Stream
40mOne SQL to Rule Them All