Mooncake: Real-Time Apache Iceberg Without Compromise
summary
Mooncake adds a real‑time layer on top of Apache Iceberg to get fresh Postgres changes into analytics with subsecond ingestion by buffering writes in Arrow and taking lightweight 500 ms snapshots while committing Iceberg snapshots infrequently (~5 minutes). It converts equality deletes into deletion vectors using a disk‑optimized hash index (index‑LSM) so reads are efficient, supports streamed large transactions and non‑blocking compaction, embeds analytics via PG Mooncake + DuckDB, reports up to 1000× faster analytics than raw Postgres, and is being absorbed into Databricks.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:03Composable HTAP vision
- 3:12OLTP vs OLAP divergence
- 8:57Postgres and lakehouse
- 12:11Mooncake components
- 13:21Real-time ingestion challenges
- 17:57Postgres replication to Iceberg
- 22:36Moonlink streaming inserts
- 26:52Handling updates and deletes
- 32:15Large transaction support
- 34:04Initial copy and recovery
- 37:36Compaction and clustering
- 41:34Fast analytics in Postgres
- 45:36Embedding DuckDB
- 49:30Iceberg mirrors in Postgres
- 51:17Session-level consistency
- 53:03Modular design
- 54:28Future directions
- 56:39Q&A: Mooncake middleware
- 58:47Q&A: HTAP comparison
- 1:00:33Q&A: Real-time drivers
related talks
37mApache Iceberg: An Architectural Look Under the Covers
21mAnalytics in Postgres –A decade in the making
05mMaking Iceberg walk and talk like Postgres
41mTip of the Iceberg
52mApache Iceberg - An Architectural Look Under the Covers
20mWrites, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™
34mApache Iceberg.
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™