Too Big for DAG Factories?
summary
Claims Airflow can handle thousands of DAGs if import-time bottlenecks are removed and DAGs are prebuilt. Uses Apache Airflow with Docker Compose, Postgres metadata DB, Redis/Celery or Kubernetes executors, JSON config files, Azure App Configuration and Terraform/Jinja for infra; dynamic DAG factory hits a 30 second dagbag import timeout and slows with API calls or heavy imports. Solution is to generate one file per DAG (static files), precompile .pyc, build into the container and CI/CD deploy — proven on 10,000 DAGs (≈17–18 minutes to load on a laptop) and works.
AI-written summary · about
chapters (7)
- 0:00Introduction to Airflow
- 4:04Airflow architecture
- 7:32The dynamic DAG challenge
- 9:15Implementation and demo
- 15:26Scaling and timeout issues
- 21:36Static generation solution
- 27:08Optimization and conclusions
related talks
35mMaximizing Airflow: Advanced Techniques for Data Pipeline Orchestration
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
18mHow we deleted 10,000 lines of code and got control of our Airflow DAGs
19m4 tricks that help you leverage your Airflow pipelines
52mBetter ETL with Managed Airflow in ADF
22mOrchestrate your data pipelines using Apache Airflow
34mData Ingestion in Python using Apache Airflow
36mApache Airflow @ Umuzi.org