Using Modern Application Principals to Automate Your Apache Airflow Data Pipelines
summary
Claims a reproducible Airflow pipeline workflow using CI/CD to build versioned container images and git-sync sidecar or shared volumes for DAG delivery, compares self-managed vs MWAA and argues total cost matters. Demos infrastructure-as-code with Terraform/CDK/ Pulumi, shows using an Airflow constraints file plus pre-commit hooks and local-runner or VS Code plugins to catch dependency hell early, and publishes code and Helm/ Docker examples that actually build and deploy repeatable environments.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 3:32Modern app principles
- 6:22Automating infrastructure
- 12:22Custom Airflow images
- 17:15Separation of concerns
- 18:40Version control for workflows
- 20:27Python dependencies
- 22:39Testing workflows
- 24:22Containerizing workflows
- 26:06Developer experience
- 28:13Airflow development tools
- 31:04Automated testing
- 33:52Fully automated deployment
- 34:13Call to action
related talks
36mElegant data pipelining with Apache Airflow
46mGetting Started With Data Pipelines Using Apache Airflow
38mUse Apache-Airflow to Build Data Workflows
26mGo with the Flow: Automating Your Workflows with Airflow
22mOrchestrate your data pipelines using Apache Airflow
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
31mThe Silent Symphony: Keeping Airflow's CI/CD and Dev Tools in Tune
16mBridging Data Science and Data Infrastructure with Apache Airflow