Serving a multi-platform Data Lake via Delta, Iceberg, Unity and Glue
summary
A rearchitected Orion platform puts S3 as single copy storage serving Delta Lake files with Glue as engineering catalog, Databricks for compute, Airflow orchestration and Unity + Snowflake for presentation, using shallow clones for synchronous inline DQ and metadata-driven onboarding. Uniform is enabled to publish Delta as uniform iceberg metadata so Snowflake reads without copying data. Result: 3.5 PB served, ~400 tables, ~80 auto-generated DAGs, 6,000+ users and cost to process a GB drops from $0.23 to about $0.08 (≈2.5× savings).
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:09Vega to Orion evolution
- 6:24Databaseless architecture
- 9:14Orion design principles
- 12:41Delta Lake features
- 13:03Orion workflow overview
- 14:52Data quality with shallow clones
- 18:47Delta Lake with Snowflake
- 20:34Uniform for lakehouse interop
- 22:19Uniform in Orion
- 24:26New architecture benefits
- 25:54Cost and scalability results
- 27:19Q&A
related talks
16mThe Evolution of Delta Lake
28mBuilding A Governed Data Lake
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
58mBuild reliable data lakes with Delta Lake & Databricks
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
25mHow to build a platform for AI and analytics based on Apache Iceberg
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
33mThe advent of the open data lake