DuckDB Meets Data Lakes
summary
DuckDB now reads Delta and Iceberg, ships a new DuckLake format and plans a 1.0 release after a DuckLake v0.3 preview; benchmarks claim astonishing single‑node scale — TPCH 1 TB on a €300 Raspberry Pi, laptop handling 10 TB and a 100 TB experiment — and a lightweight min/max index per ~100k row groups that cuts IO. Small‑insert handling moves into a catalog DB to enable multiplayer duck DB workflows and features like trailing commas and GROUP BY ALL copied by five major systems.
AI-written summary · about
chapters (8)
- 0:00DuckDB and data lakes
- 11:35DuckDB's origin
- 17:56DuckDB performance
- 24:25Formats protocols integrations
- 26:35Limitations and data lakes
- 29:47Data lakes and formats
- 32:58DuckLake architecture
- 37:14DuckLake production
related talks
26mBuilding Data Lake Using DuckDB
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
47mWhy DuckDB could be the engine for your Data Lake?
28mAnalyze data easily with duckdb - and the implications on data architectures
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
51mMotherduck: Data Analytics in the Post Big-Data Era
27mState of the Duck
05mAnnouncing DuckDB Support for Delta Lake and a DuckDB Extension to Unity Catalog