Getting Started With Data Pipelines Using Apache Airflow
summary
Shows a working Apache Airflow pipeline defined as Python DAGs, using bash, Python, S3 sensors, Athena and Redshift operators, run locally with a provided local-runner and later deployable to Amazon MWAA. Demonstrates failure recovery: Athena tasks fail (missing DB), create the DB, clear and retry just the failed tasks. Shows state passing via XCom pushes and pulls query IDs and a toy transform that removed every "e" from movie titles to prove the pipeline actually runs end to end.
AI-written summary · about
chapters (8)
- 0:00Apache Airflow introduction
- 3:47Why Airflow matters
- 8:43Core workflow concepts
- 15:50Airflow architecture
- 23:57Simple workflow demo
- 31:32Real-world data pipeline
- 38:21Recent updates & community
- 41:09Q&A on testing & dependencies
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
36mElegant data pipelining with Apache Airflow
16mBridging Data Science and Data Infrastructure with Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
35mUsing Modern Application Principals to Automate Your Apache Airflow Data Pipelines
14mScheduling machine learning pipelines using Apache Airflow
01h 18mIntroduction to orchestration ETL processes with Apache Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow