Delta Live Tables: Building Reliable ETL Pipelines with Azure Databricks
summary
Shows how Delta Live Tables on Azure Databricks builds medallion ETL with concrete mechanics: bronze/silver/gold tables, Auto Loader streaming for incremental file pickup, and an apply_changes merge that performs inserts/updates/deletes. Demo uses NYC yellow‑taxi parquet/CSV data, auto-detects dependencies and lineage across notebooks, and reports 0.8% rows dropped by declared constraints. Pipelines auto-scale, persist raw history in bronze, and the demo pipelines run successfully with logs and expectation files available.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:24Mohit Batra
- 2:43Delta Architecture Layers
- 8:03Complexity in ETL Pipelines
- 11:09Pipelines with Delta Live Tables
- 11:42Components
- 13:28Demo 1: Simple Pipeline
- 24:37Demo 2: Complex & Streaming Pipeline
- 38:19Session Evaluation
related talks
49mBuilding Advance Analytics pipelines with Azure Databricks
29mML in Production on Databricks
31mSimplifying upserts and deletes on Delta Lake tables
44mDeep-Dive into Delta Lake
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
38mGetting Started with Delta Lake
31mDelta Lake: Open Source Reliability and Quality for Data Lakes
55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?