Apache Iceberg Planning Explained
summary
Apache Iceberg stores table state as a persistent tree of root metadata, snapshots, manifest lists and manifests so planners filter metadata before touching data. Benchmarks use a table with 10 million partitions and 20 million data files (15 cols, bounds on 6), producing ~850 manifests (~7 GB). Local planning finds partition-aligned queries in ~0.25 seconds, distributed planning finds handful of files in ~2.5s but full scans hit Spark serialization (~20s) overhead. Adding 10M delete files (68 delete manifests, ~500 MB) costs seconds for delete indexing; local planning remains fast for partition-aligned queries.
AI-written summary · about
chapters (11)
- 0:00Introduction to Apache Iceberg
- 0:31Key metadata concepts in Iceberg
- 3:00Metadata evolution and structural sharing
- 5:49Planning overview without delete files
- 8:39Two-stage planning process explained
- 10:27Benchmarking planning performance
- 12:11Highly selective query results
- 13:18Distributed planning performance
- 15:25Planning with delete files
- 18:40Performance with delete files
- 20:28Key takeaways and summary
related talks
52mApache Iceberg - An Architectural Look Under the Covers
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
13mExploring the Depths of Apache Iceberg's Metadata Capabilities
34mApache Iceberg.
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
41mTip of the Iceberg
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
49mHow Apache Iceberg Really Works