Apache Iceberg - An Architectural Look Under the Covers
summary
Explains how Apache Iceberg replaces folder-based tables with a metadata tree (catalog → metadata file → manifest list → manifests → data files) and uses per-file min/max column stats and hidden partitioning to prune work. Gives concrete numbers from Netflix: a dataset with 2.7 million files in 2,688 partitions sees planning time fall from 9.6 minutes to 10 seconds and full query times drop to seconds once min/max pruning is applied. Shows Spark/Flink/Dremio integrations, Project Nessie for git-like catalogs, ACID snapshots, time travel, and notes compaction must be scheduled externally.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:06Table formats and Iceberg
- 10:52Netflix’s need
- 14:46Iceberg architecture
- 21:49Creating Iceberg tables
- 25:18Inserting data and snapshots
- 27:04Upserts and merges
- 28:51Reading data and queries
- 30:37Time travel with snapshots
- 32:20Netflix’s performance gains
- 36:36Key features
- 41:09Optimizing Iceberg tables
- 42:34Project Nessie features
- 45:02Q&A
related talks
37mApache Iceberg: An Architectural Look Under the Covers
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
34mApache Iceberg.
13mExploring the Depths of Apache Iceberg's Metadata Capabilities
22mApache Iceberg Planning Explained
01h 04mAn Extremely Technical Overview of How Apache Iceberg Planning Actually Works
41mTip of the Iceberg