Unbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
summary
Argues that open data lake formats plus fast analytical DBs replace monolithic cloud warehouses by example: Apache Iceberg / Delta Lake / Hudi as catalogs, ClickHouse as the hot real‑time layer, and a common double‑write pattern (hot DB for quick inserts, lake for cold storage). Cites ~880,000 Python packages and 2.28 trillion daily downloads in a demo, 8k contributors / 15k Slack members for ClickHouse, and a Sloan Kettering deployment that runs LLM agents (Anthropic + Bedrock) querying terabytes via SQL with materialized views for speed.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:06Data warehousing evolution
- 5:14Data lakes and formats
- 10:07Iceberg vs Delta vs Hudi
- 13:39Analytical databases
- 21:43Combining data lakes with analytical databases
- 26:36AI agents and open lakes
- 36:31Warehouses vs data lakes
- 38:15Closing
related talks
32mHow open source is re-shaping the cloud data warehouse landscape
28mRethinking Status Quo: Open Source and the Cloud Data Warehouse
25mBuild open format data lakehouse
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
23mThe Open Source Analytic Universe, Version 2022
34mData Lakehouse: Who, Why and How
41mOn Data: Sky above, sand below, peace within
31mData Warehouses Meet Data Lakes