Better ETL with Managed Airflow in ADF

on YouTube ↗52m

" summary

Managed Airflow in ADF brings Apache Airflow as a PaaS inside Data Factory, with demos showing DAGs that run ADF pipelines asynchronously (async operator + sensor) and Databricks notebook-triggering via connection IDs; Git sync pulls Python DAGs from GitHub, but startup is slow (20–25 minute startup), secrets/key vault integration is brittle and env vars appear plain text, CLI/DevOps support is limited, autoscale exists but you can't pause or turn off the service, and the preview surface is change-prone.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:02What is Apache Airflow
3. 5:18Managed airflow in ADF
4. 7:08ADF vs. Airflow
5. 9:08Benefits of ETL as code
6. 12:16Understanding DAGs and tasks
7. 23:39Demo and configuration
8. 47:00Conclusion and considerations

" related talks

• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 50mAzure Data Factory patterns and best practicesShirley Wang · SQLBits 2020
• 46mGetting Started With Data Pipelines Using Apache AirflowRicardo Sueiras · Big Data Conference Europe 2022
• 34mData Ingestion in Python using Apache AirflowSowmya Ganapathi Krishnan, Yew Wee · PyCon APAC 2018
• 36mElegant data pipelining with Apache AirflowBolke de Bruin · PyData Amsterdam 2018
• 35mUsing Modern Application Principals to Automate Your Apache Airflow Data PipelinesRicardo Sueiras · Big Data Conference Europe 2023
• 16mBridging Data Science and Data Infrastructure with Apache AirflowDaniel Imberman · PyCon US 2020