Apache Iceberg.
summary
Apache Iceberg replaces Hive’s directory-level tracking with file‑level tracking, storing table state in a catalog plus metadata files, manifest lists and manifests that summarize Parquet file stats for fast pruning; engines like Spark, Trino, Flink and DuckDB plug into the format on S3 or cloud object stores. It provides snapshots, time travel and atomic swaps via catalogs (Glue, Hive metastore, Project Nessie), supports copy‑on‑write or merge‑on‑read writes, and Netflix reports query planning dropping from ~10 minutes to ~42 seconds. manifest files and catalog snapshots are the core performance levers.
AI-written summary · about
chapters (9)
- 0:00Introduction to Apache Iceberg
- 7:27Evolution of data architectures
- 15:24Problems with Hive table format
- 17:32What is Apache Iceberg
- 18:34Iceberg architecture deep dive
- 24:05Query examples and time travel
- 27:54Choosing a table format
- 29:19Interoperability with OneTable
- 30:21Q&A session
related talks
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
37mApache Iceberg: An Architectural Look Under the Covers
52mApache Iceberg - An Architectural Look Under the Covers
27mHive-Iceberg - Breaking the Ice: A Closer Look at Hive-Iceberg
41mTip of the Iceberg
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
50mDesigning a scalable data platform based on Apache Spark/Iceberg