Managing Python Dependencies at scale
summary
Apache Airflow manages ~80 packages (about 630–650 transitive dependencies, 70+ provider packages) by generating a golden set of pinned dependencies and publishing constraints files pulled from GitHub so installs stay reproducible. CI runs a CI Canary that auto-upgrades and tests candidate dependency bumps, commits new constraints only when all tests pass, and provides Docker dev images; the system scales to ~20 upstream updates/week and successfully automates upgrades and reproducible builds.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:56What Apache Airflow is
- 4:48Python dependencies basics
- 10:08Dependency management approaches
- 14:20Airflow’s dependency challenges
- 15:03Maintainer’s view on dependencies
- 17:31Deployment needs for consistency
- 18:57User flexibility vs consistency
- 20:21Contributor workflow issues
- 22:11Airflow’s pip constraints
- 26:24Key takeaways
- 27:04Q&A
related talks
31mNo more dependency nightmares: the Wix way for a healthy and stable python platform
31mThe Silent Symphony: Keeping Airflow's CI/CD and Dev Tools in Tune
30mreliable testing with pip constraints
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
07mLightning Talk – Python Dependencies the Right Way
42mSo much depends upon... a python app's dependencies
37mAutomated deployment of Python packages for development
35mUsing Modern Application Principals to Automate Your Apache Airflow Data Pipelines