pg_lake: Postgres as a lakehouse
summary
pg_lake adds Iceberg tables to Postgres, implements a separate DuckDB delegation server (avoiding per-connection memory blowup), and provides pg_incremental for once‑only incremental pipelines. It creates Iceberg metadata and Parquet files, runs compaction every 10 minutes, delegates analytical queries to DuckDB via the Postgres protocol with a file cache, and supports importing/exporting CSV Parquet JSON and wildcard foreign tables. Batch ETL runs every minute or 10 seconds and produces production‑ready lakehouse workflows with transactional guarantees.
AI-written summary · about
chapters (11)
- 0:00Music & introduction
- 0:37Lakehouse concept definition
- 2:40Apache Iceberg explained
- 3:43How pg_lake works with PostgreSQL & history
- 5:59PostgreSQL demos: files, queries, & exports
- 8:05Iceberg internals & transactions
- 13:02Fast analytics with DuckDB
- 14:37Building pipelines with pg_incremental
- 16:25Examples: archiving, event queues, & auto-load JSON logs
- 21:31Connecting to warehouses & ecosystems
- 23:21Key takeaways and closing
related talks
27mBuilding a PostgreSQL data warehouse
05mMaking Iceberg walk and talk like Postgres
34mAnalytics and AI with PostgreSQL - Mission Possible!
26mBuilding Data Lake Using DuckDB
34mAnalytics and AI with PostgreSQL - Mission Possible!
40mDuckDB Meets Data Lakes
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
21mWhat can Postgres learn from DuckDB?