DataEngBytes 2023 - SYD-T3-01 - Scott Eade
summary
Delta Lake 3.0 writes Iceberg-format metadata alongside Delta metadata so tools that speak Iceberg (BigQuery external tables, Snowflake private preview) can read the same parquet files; enabling UniForm is a table property and costs <5% write overhead. Demo shows creating a Delta table with the property, exposing the Iceberg metadata path on GCS and querying it from BigQuery. Requires runtime 13.2+, uses Delta kernel and is Iceberg-only today.
AI-written summary · about
chapters (8)
- 0:00Delta Lake 3.0 intro
- 3:53Delta Lake as open source
- 7:02Liquid clustering
- 11:34Delta Kernel
- 13:41Uniform for cross-format
- 18:42Enabling uniform
- 20:09Uniform demo
- 24:47Closing & Q&A
related talks
21mDataEngBytes 2023 - MEL-T1-05 - Fabio Ramos
16mThe Evolution of Delta Lake
28mDataEngBytes 2023 - PER-T1-04 - Nathan Glover
29mServing a multi-platform Data Lake via Delta, Iceberg, Unity and Glue
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
05mThe Future of Lakehouse Format Interoperability