How Apache Iceberg Enables Multi-Engine Data Platforms
summary
Apache Iceberg decouples storage from compute so Spark, Trino and Flink read and write the same table using metadata-driven concurrency and snapshot-isolation via a metadata.json + manifest chain; writers upload files in parallel and complete with an atomic catalog pointer swap (optimistic commit). Operational guidance covers file sizing (aim ~128–384 MB), compaction cadence (hourly for streaming, weekly for batch), snapshot expiration, and cleaning orphan files. Cloudera adds a REST catalog and a closed-source Lakehouse Optimizer claiming ~40–80% storage reduction and 2–4x query speedups.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker context
- 2:32Problem of multi-engine data copies
- 5:45What is Apache Iceberg
- 8:13Iceberg metadata chain explained
- 12:52Engine selection for Iceberg
- 16:40Cloudera’s Iceberg integration
- 18:26Cloudera’s open data lakehouse
- 22:18Migrating from Hive to Iceberg
- 23:21Operational best practices for Iceberg
- 26:56Common Iceberg pitfalls
- 28:45Key takeaways and conclusion
related talks
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
34mApache Iceberg.
37mApache Iceberg: An Architectural Look Under the Covers
52mApache Iceberg - An Architectural Look Under the Covers
50mDesigning a scalable data platform based on Apache Spark/Iceberg
41mTip of the Iceberg
49mHow Apache Iceberg Really Works
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™