Introducing Apache Iceberg: Tables Designed for Object Stores
summary
Apache Iceberg replaces Hive's directory-based tables with a metadata layer of immutable snapshots and manifest files so writers produce atomic commits and readers use snapshot isolation without locking. Netflix benchmarks show planning drops from 9.5 minutes (400,000 splits) to ~10 seconds (15,000 splits) and with file-level stats to 25s planning and 42s wall time. Iceberg avoids no S3 directory listings, records per-file stats and min/max bounds for partition and bucket pruning, supports Avro/ORC/Parquet, Spark/Presto/Hive, and uses column IDs for schema evolution and copy-on-write metadata for safe rollbacks and GC.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:00Performance
- 3:05Table Format
- 7:35Metastore
- 8:37Hive Acid
- 9:22Hive Bucketing
- 14:17Iceberg
- 15:20Copyonwrite
- 16:10Manifest Files
- 17:20Tracking Data
- 18:05Updating Manifests
- 18:44Conflict Detection
- 19:28Merge Files
- 27:38Other projects
- 28:03Getting started
- 28:28Github support
- 29:11Future work
- 29:48Delta files
- 30:58Questions
related talks
34mApache Iceberg.
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
52mApache Iceberg - An Architectural Look Under the Covers
41mTip of the Iceberg
27mHive-Iceberg - Breaking the Ice: A Closer Look at Hive-Iceberg
13mExploring the Depths of Apache Iceberg's Metadata Capabilities
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
50mDesigning a scalable data platform based on Apache Spark/Iceberg