Design Decisions Behind Cube Store - a Blazingly Fast Open-Source Rollup Storage Layer
summary
Cube Store is an open-source rollup storage layer that materializes pre-aggregated tables and serves queries with 200 ms p95 target by combining streaming in‑memory chunks with minute‑level Parquet flushes. It uses Apache Arrow and DataFusion for execution, stores Parquet on object stores, and keeps metadata in RocksDB metastore. Shared‑nothing workers own partitions, perform collocated and calculated joins without reshuffles, scale to billions of input rows, and aim to deliver sub‑second latency and high concurrency.
AI-written summary · about
chapters (17)
- 0:00Start
- 2:51User-facing analytics
- 3:43Uber Eats
- 5:51Pipelines: pros & cons
- 8:46Time series databases
- 9:13TSDBs: pros & cons
- 13:19OLAP datastores: pros & cons
- 15:23Cloud data warehouses
- 16:38Cloud DWHs: pros & cons
- 19:27Enter Cube Store
- 21:40Requirements
- 23:50Cube Store Architecture
- 29:34Design decisions matrix
- 31:35Routing for in-memory chunks
- 33:15Shared-nothing architecture
- 35:12Lambda architecture query
- 36:26Collocated join indexes
related talks
42mColumnar Storage: Redefining Data Management for the Modern Era
33mThe advent of the open data lake
26mColumnar Storage: Redefining Data Management for the Modern Era
37mKEYNOTE: Data Architecture Turned Upside Down
33mBuilding an Elastic, Resilient Data Layer to Power New Retail Experiences
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
26mPetabyte Scale Datalake Table Management with Ray, Arrow, Parquet, and S3
29mCubeFS in Action: Empowering Users Through Case Studies