Getting Started with Apache Airflow - Building Your First Workflow
summary
Clear, concrete outcome: a runnable DAG that uploads jokes to S3 and MySQL on a daily schedule, demonstrated successfully in Airflow 2.7.2 running via a local Docker-based runner. Shows scheduler behavior, catchup/backfill pitfalls (thousands of backfilled runs when misconfigured), executor tradeoffs and the new task flow API and defer operators for async work, uses XCom and the metadata DB for state, and even converts code with generative AI (Anthropic Claude) during the demo.
AI-written summary · about
chapters (17)
- 0:00Apache Airflow introduction
- 1:32Why Airflow was created
- 3:18Airflow as workflow orchestrator
- 7:33Airflow architecture overview
- 17:54Task Flow API introduction
- 18:39Building first workflow
- 21:07Demo environment setup
- 22:51Workflow tasks and operators
- 24:18DAG for joke repository
- 27:53Connections and DAG run
- 29:42Deploying the DAG
- 30:23Monitoring workflows
- 31:47Workflow results review
- 32:51Workflow recap
- 33:34Running Airflow on AWS
- 34:18Contributing to Airflow
- 35:01Airflow learning resources
related talks
38mUse Apache-Airflow to Build Data Workflows
32mOrchestrating Your Data Pipelines with Apache Airflow
48mFlow Is in the Air: Best Practices with Apache Airflow
01h 25mApache Airflow in the Cloud: Programmatically orchestrating workloads w/ Py
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
32mBuilding Analytics Workflow using Airflow and Spark
36mElegant data pipelining with Apache Airflow