The Evolution of Delta Lake
summary
Announces Delta Lake UniForm GA — a universal format that reads Delta as Iceberg or Hudi with minimal overhead, already moving over four exabytes of data and claiming nine exabytes processed (daily) and over a billion clusters/year; community has 500+ contributors and ~66% non‑Databricks commits. Ships a small Delta Kernel library, a DuckDB extension with Unity Catalog support, deletion vectors saving ~100 trillion row rewrites, and Liquid Clustering (writes up to 7x, reads 12x faster) plus a JSON variant type ~8x faster than strings, all landing in Delta 4.0 and Spark 4.0 code.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:10Uniform format interoperability
- 4:03Ecosystem and community growth
- 5:25DuckDB integration
- 9:45Community extensions and plans
- 11:12Delta Lake 4.0 features
related talks
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
32mDesigning Data Infrastructure in the Age of Generative AI
38mGetting Started with Delta Lake
29mServing a multi-platform Data Lake via Delta, Iceberg, Unity and Glue
20mDelta Lake: Turn your Data Lake into a Lakehouse
28mLakehouse Architecture with Delta Lake and Databricks
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
39mDelta Lake: Diving into Data Lakes Without the Downsides