Declarative ETL pipelines with Delta Live Tables

on YouTube ↗51m

" summary

Declarative pipeline framework that turns SQL or Python notebooks into production ETL with cloud_files autoloader for incremental file ingest, exactly-once ingestion semantics, automatic exponential-backoff retries and schema-change restarts. Pipelines declare table dependencies and lineage, support streaming or batch, run autoscaling clusters, and use apply_changes for CDC→SCD to convert change logs into type‑1/type‑2 tables. Operational metadata and run metrics are emitted as an event log as Delta table. Limitations: no multi‑table atomic commits and some runtime feature lag.

AI-written summary · about

" chapters (14)

1. 0:00Intro
2. 5:57What is a Streaming Live Table? Based on Spark™ Structured Streaming
3. 10:07Development vs Production Fast iteration or enterprise grade reliability
4. 14:08Choosing pipeline boundaries Break up pipelines at natural external divisions.
5. 19:16Pitfall: hard-code sources & destinations Problem: Hard coding the source & destination makes it impossible to test changes outside of production, breaking CI/CD
6. 22:08Ensure correctness with Expectations Expectations are tests that ensure data quality in production
7. 23:37Expectations using the power of SQL Use SQL aggregates and joins to perform complex validations
8. 25:09Using Python Write advanced DataFrame code and UDFs
9. 30:21Installing libraries with pip pip is a package installer for python
10. 33:14Best Practice: Integrate using the event log Use the information in the event log with your existing operational tools.
11. 33:44DLT Automates Failure Recovery Transient issues are handled by built-in retry logic
12. 35:51Modularize your code with configuration Avoid hard coding paths, topic names, and other constants in your code.
13. 39:04Workflow Orchestration For Triggered DLT Pipelines
14. 41:10Use Delta for infinite retention Delta provides cheap, elastic and governable storage for transient sources

" related talks

• 22mReliable data engineering made easyMahdi Askari · DataEngBytes 2021
• 21mBuilding Data Integration Services In Real TimeKerry McRae · DataEngBytes 2020
• 45mThe Data Lakehouse: A tech talk for everyone who loves dataFrank Munz · Voxxed Days Ticino 2023
• 38mGetting Started with Delta LakeGuenia Izquierdo Delgado, Sajith Appukuttan · Open AI + Data Forum @ Open Source Summit North America 2023
• 30mThe Incremental ETL ArchitectureJohn O'Dwyer · Scale By the Bay 2021
• 31mSimplifying upserts and deletes on Delta Lake tablesPrashanth Babu · Berlin Buzzwords 2021
• 04mInterviewMichael Paul Armbrust · Scale By The Bay 2019
• 20mDelta Lake: Turn your Data Lake into a LakehouseVinoaj Vijeyakumaar · DataEngBytes 2022