Tip of the Iceberg
summary
Apache Iceberg is an open table specification that lets SQL-like ACID operations run directly on object storage, using manifest lists and manifests to provide snapshotting and time travel. The demo uses PyIceberg and the NYC taxi data, where a schema/partition change reads 86 files and rewrites about 1500 files, revealing a data-quality mismatch (January shows 40 days not 31). Iceberg uses per-file column stats for pruning, supports lazy schema/partition evolution, and adds multi-table transactions and positional deletes.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:30What is Iceberg
- 4:27Data architecture
- 5:54Why Iceberg
- 7:50Query Complexity
- 9:52Iceberg Table
- 13:06Iceberg Practice
- 18:59Python Implementation
- 23:07Summary
- 24:09Questions
- 26:21Transactional processing
- 27:52Query planning
- 30:14Partition size optimization
- 32:37Is there a way to do that
- 33:49Manifest lists
- 35:20GDPR compliance
- 36:41Preventing raw data
- 37:48Listing of data
- 39:07Partitioning
related talks
01h 12mHands-on with Apache Iceberg
37mApache Iceberg: An Architectural Look Under the Covers
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
52mApache Iceberg - An Architectural Look Under the Covers
49mHow Apache Iceberg Really Works
28mBridging Python and Apache Iceberg: The Power of PyIceberg