Open-Source OLAP on Open-Source Data Warehouses.
summary
Combines Iceberg on S3 as an ACID, cheap partitioned data lake with OLAP engines to get subsecond analytics by caching partitions; Delta Lake benchmarks run about 10 to 30% faster than Iceberg but Iceberg stays portable. Implements a lean partition‑sync: S3 metadata updates -> SQS -> refresh OLAP cache so hot data is instantly queryable. Notes StarRocks caches raw partition files locally while ClickHouse mostly caches final results, OLAP is fast but has eventual consistent writes, and the Iceberg Python library lets this run without heavy AWS resources.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:35Shea Hawkins Introduction
- 1:31Data Scientists
- 2:31Competitors
- 4:20Iceberg vs Delta Lake
- 4:44Snowflake vs Unicorn
- 5:23Snowflake
- 6:12OLAP
- 8:02Data Lakes vs OLAP
- 9:22OLAP Weaknesses
- 13:08Cache Setups
- 13:44More Complex Cases
- 14:08Partition Sync
- 16:15Iceberg Python
- 16:55Summary
related talks
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
41mTip of the Iceberg
25mBuild open format data lakehouse
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
36mArcticDB, the OLAP antidote
32mHow open source is re-shaping the cloud data warehouse landscape
32mLow Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...