Bridging Python and Apache Iceberg: The Power of PyIceberg
summary
PyIceberg is a pure Python Apache Iceberg client that actually writes Parquet, builds manifests and manifest lists, and performs atomic metadata swaps so table updates succeed with optimistic concurrency. It reads via PyArrow, Pandas, Polars and prunes files using manifest stats, so simple one-line append/overwrite/upsert calls generate snapshots and metadata behind the scenes. It supports schema evolution, hidden partitioning, and time travel and demonstrates these features with runnable code examples that successfully create, query, and restore table states.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:41Apache Iceberg overview
- 2:50Python data workflow challenges
- 4:37How Apache Iceberg solves these challenges
- 7:05Iceberg structure
- 10:13PyIceberg introduction
- 11:59Working with Iceberg tables
- 14:53PyIceberg read optimizations
- 18:10PyIceberg write operations
- 19:55Schema evolution
- 20:38Updating partitions
- 21:20Transactions in PyIceberg
- 22:22Time travel queries
- 23:26Cross-tool compatibility
- 25:12Q&A
related talks
41mTip of the Iceberg
01h 12mHands-on with Apache Iceberg
37mApache Iceberg: An Architectural Look Under the Covers
46mTake the Polar Plunge: A Fearless Introduction to Apache Iceberg™
34mApache Iceberg.
49mHow Apache Iceberg Really Works
01h 04mAn Extremely Technical Overview of How Apache Iceberg Planning Actually Works
52mApache Iceberg - An Architectural Look Under the Covers