Building Data Lake Using DuckDB
summary
A managed data lake pipeline that uses DuckDB per Kubernetes pod to perform distributed Parquet rewrites and CDC-style merges into Iceberg/Delta on S3 and ADLS is implemented: writers split large loads into parallel tasks, worker servers download Parquet, run SQL upsert/delete/update via DuckDB, write new Parquet files and update metadata. Benchmarks show parallel merges scale until memory/IO limits and performance saturates at ~12 threads for this workload; end-to-end download/upload often uses AWS SDK because AWS SDK faster for downloads in tests. Ongoing work: remote-only queries, encryption, output-size thresholds.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:28Fan as data movement
- 5:20Why build managed lake
- 8:56Data lake architecture
- 13:55DuckDB for lake ops
- 20:48Why DuckDB chosen
- 25:00Future explorations & Q&A
related talks
06mA quack at building scalable data pipelines with DuckDB
40mDuckDB Meets Data Lakes
28mAnalyze data easily with duckdb - and the implications on data architectures
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
55mDuckLake: Learning from Cloud Data Warehouses to Build a Robust “Lakehouse”
47mWhy DuckDB could be the engine for your Data Lake?
01h 02mAnalytics for not-so-big data with DuckDB
40mDuckDB beyond the notebook