Maximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
summary
Company runs about 500 DAGs and handles roughly 5 million tasks monthly, mostly on AWS (EMR/EMR Serverless, Sagemaker, Databricks). Common failures include scheduler lockups from massive UI/DB queries (fixed by killing blocking queries), long‑running heavy operators tying up pods, and sensors in poke mode exhausting workers. Recommendations: use reschedule sensors, separate pools for greedy DAGs, orchestrate heavy work off Airflow (EMR Serverless / AWS Batch) and minimize scheduler footprint to keep scheduling timely.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker intro
- 1:51Foursquare orchestration tools overview
- 5:46Scale of Foursquare’s data pipelines
- 7:15Why Airflow was chosen
- 9:50Common Airflow pitfalls
- 12:42Misuse of slots and pools
- 15:09Sensor scheduling best practices
- 18:23Avoiding heavy operators
- 20:32Greedy DAGs and resource limits
- 22:16Scheduler heartbeat failure case
- 25:08Designing efficient DAGs
- 27:40Moving to Astronomer
- 29:03Airflow extensibility and misuse
- 30:49Minimizing scheduler workload
- 31:09Q&A session
related talks
22mOrchestrate your data pipelines using Apache Airflow
26mModern Data Orchestrators
56mWhy Airflow is a secret ingredient in MLOps
45mDefining data pipelines
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
43mHelp! My data is late!
38mToo Big for DAG Factories?