Orchestrate your data pipelines using Apache Airflow
summary
Overview of Apache Airflow with concrete tradeoffs: when to keep cron jobs for simple one-off schedules and when to migrate for dependency, retry and monitoring needs; explanation of core components like DAGs, operators, scheduler and the metadata database; shows a live demo building a small ETL DAG, triggering runs, viewing logs and retries in the web UI, and claims this replaces brittle shell scripts for multi-step batch pipelines while keeping simple tasks on cron.
AI-written summary · about
chapters (4)
- 0:00Introduction to Airflow
- 6:00Data pipeline concepts
- 12:00Setting up Airflow components
- 18:00Running a sample workflow
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
36mElegant data pipelining with Apache Airflow
35mBecome a Conductor with Apache Airflow
46mGetting Started With Data Pipelines Using Apache Airflow
39mSell cron, buy Airflow: Modern data pipelines in finance
34mData Ingestion in Python using Apache Airflow
45mDefining data pipelines
16mBridging Data Science and Data Infrastructure with Apache Airflow