Table Formats Change Everything (By Not Changing Anything)
Talkat Open AI + Data Forum @ Open Source Summit North America 2022 · on Wednesday, 2022-06-22· at 14:35· Room 408/409 (Level 4), JW Marriott Austin, 110 E 2nd Street Austin, TX 78701·
summary
Table formats (Delta Lake, Apache Iceberg, Apache Hudi) pack Parquet files under a metadata layer that exposes ACID updates and concurrent writers via JSON snapshot logs, optimistic commits (first writer wins, others retry), and file-level manifests. They add time travel and zero-copy clones so models can be retrained against exact historical data, store file statistics (min/max) to speed queries, and integrate with Spark/Databricks and cloud object stores like S3. Trade-offs: more metadata to store and manage, need vacuum/compaction, and cold‑storage versioning is not standard yet.
AI-written summary · about
chapters (4)
related talks
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
24mData Mesh and Lakehouse
33mThe Pace of Innovation in Delta Lake
39mIntroducing Apache Iceberg: Tables Designed for Object Stores
33mThe advent of the open data lake
28mLakehouse Architecture with Delta Lake and Databricks
44mSolving the world's toughest problems
38mGetting Started with Delta Lake