Running Apache Airflow at Scale
summary
Shopify runs Apache Airflow with 12,000 DAGs and about 400,000 tasks per day, and describes concrete fixes: a hybrid GCS+mounted filestore sync to cut DAG deployment latency, custom DAG validators and mutators, aggressive DB truncation and MySQL query fixes, encouragement of the KubernetesPodOperator to avoid dependency hell, and a bespoke surf CLI plus multi-instance sharding that reduces blast radius but introduces metadata migration and routing edge cases that still need work.
AI-written summary · about
chapters (13)
- 0:00Speakers and Airflow at Shopify
- 1:09What is Apache Airflow
- 2:54Airflow DAGs at Shopify
- 3:55Airflow infrastructure
- 5:20Running Airflow at scale
- 8:28Scheduler performance fixes
- 10:13Multi-instance Airflow need
- 12:42Shopify’s multi-instance design
- 14:26DAG routing across instances
- 16:33Airflow UI customizations
- 17:37Surf tooling for Airflow
- 20:23Future Airflow upgrades
- 22:30Q&A
related talks
38mLessons Learned from the Migration to Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
01h 08mThe State of Apache Airflow
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
36mApache Airflow in Production - Bad vs Best Practices
06mComparing Argo Workflows and Apache Airflow at Scale
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
31mIntro to GKE setup of airflow with helm and terraform