A Practical Introduction to Airflow
summary
Airflow is presented with concrete details: a production pipeline at Clover runs about 450 tasks and uses a Postgres metadata DB, S3 for logs, and the Celery executor with RabbitMQ/Redis; pipelines are Python DAGs built from operators and YAML, can backfill, and use max_active_runs and concurrency limits (many pipelines set max_active_runs=1). Deployment requires restarting long‑running processes (use celeryd max_tasks_per_child=1); beware the confusing execution date offset.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:29What is Airflow
- 3:18Airflow pipelines with DAGs
- 7:19Airflow UI tour
- 11:18Production components
- 16:19Building a pipeline
- 24:04Task arguments and dependencies
- 29:29Executors and debugging
- 33:25Pipeline testing tips
- 36:22Deploying code
- 38:30Why Clover Health chose Airflow
- 45:18Q&A
related talks
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
36mApache Airflow in Production - Bad vs Best Practices
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
36mElegant data pipelining with Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
25mMixing reliability with Celery for delicious async tasks
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows