Become a Conductor with Apache Airflow
summary
Introduces Apache Airflow's origin and core components (webserver, scheduler, workers, Postgres metadata) and notes it’s used by 250+ companies. Demonstrates two live Docker Compose setups: a local Airflow that reads DAG Python code with an SQS emulation and a MinIO-based S3 emulation, showing sensor-based branching, custom operators, XCom for small signals, and the scheduler’s backfill/catch-up behavior. Also covers practical cautions about authentication, key encryption, and managing shared repository dependencies.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:29A propos d'Airflow
- 3:38A quoi sert Airflow
- 11:09Exemple de set up
- 24:24A propos de nos cas d'utilisation
- 26:24Retour d'expérience
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
22mOrchestrate your data pipelines using Apache Airflow
01h 18mIntroduction to orchestration ETL processes with Apache Airflow
43mHelp! My data is late!
34mData Ingestion in Python using Apache Airflow
36mApache Airflow @ Umuzi.org
56mWhy Airflow is a secret ingredient in MLOps