Introduction to orchestration ETL processes with Apache Airflow

on YouTube ↗01h 59m

" summary

Hands‑on demo shows a reproducible cluster: a Dockerized Airflow cluster with MS SQL source and Postgres target, GitHub code and 400MB backups. Concrete claims: DAGs defined in Python, use of TaskFlow XComs and SQL sensor polling every 30s, a chunked insert controlled by a variable (default 20) and parameterized runs for selected currencies. Examples build a dimension and a fact load, detect new rows and successfully insert them into the warehouse.

AI-written summary · about

" chapters (9)

1. 0:00Apache Airflow and ETL intro
2. 10:12Airflow architecture and UI
3. 27:28DAGs tasks operators sensors
4. 41:05Docker setup for tutorial
5. 1:03:09Database config for ETL
6. 1:30:47Currency dimension ETL example
7. 1:37:34Sensors for new data
8. 1:41:36Parameterized ETL with chunks
9. 1:48:01Q&A

" related talks

• 01h 18mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 24
• 35mBecome a Conductor with Apache AirflowMarwen Ben Dhahbia · Devoxx France 2021
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 46mGetting Started With Data Pipelines Using Apache AirflowRicardo Sueiras · Big Data Conference Europe 2022
• 34mData Ingestion in Python using Apache AirflowSowmya Ganapathi Krishnan, Yew Wee · PyCon APAC 2018
• 26mGo with the Flow: Automating Your Workflows with AirflowLeo Guinan · PyOhio 2018
• 16mBridging Data Science and Data Infrastructure with Apache AirflowDaniel Imberman · PyCon US 2020
• 36mElegant data pipelining with Apache AirflowBolke de Bruin · PyData Amsterdam 2018