Building a Lakehouse on the Microsoft Intelligent Data Platform
summary
Shows how to build a working lakehouse on Synapse using Spark for ETL, Delta as the file protocol with a Delta transaction log enabling ACID, time travel and table-level rollback, and serverless SQL for serving. Implements metadata-driven notebooks called with a parameter cell (PaperMill) from integration pipelines to load bronze/silver/gold layers. Notes: dedicated pools for heavy, Spark for transformations, serverless has ~10s cold start, ~£5/TB query cost, cannot control file layout when writing from serverless, and the pipeline approach scales and reliably restores data.
AI-written summary · about
chapters (15)
- 0:00Introduction to Lakehouse
- 1:44Microsoft Intelligent Data Platform
- 3:09What is Synapse?
- 5:56Synapse Compute Engines
- 8:27Spark Engine Deep Dive
- 10:35Serverless SQL Engine
- 13:23What is a Lakehouse?
- 17:12Evolution of Data Lakes
- 19:40Delta Lake Explained
- 24:35Building a Lakehouse in Synapse
- 27:44Modern ETL with Spark
- 33:43ETL Process in Lakehouses
- 36:57Serving Data with Serverless
- 40:50Power BI Integration
- 44:00Governance with Purview
related talks
42mBuilding a Lakehouse data platform using Delta Lake, PySpark, and Trino
28mLakehouse Architecture with Delta Lake and Databricks
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake
38mGetting Started with Delta Lake
37mBuild a Modern Data Platform on AWS
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
25mBuild open format data lakehouse
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales