Help! My data is late!
summary
Airflow is presented as an open-source pipeline orchestrator used for data workflows, available both self-hosted and as managed services from cloud providers. The talk explains Airflow’s architecture and core concepts, how it integrates into data environments including multi-cloud operators, and shares practical experience and recommended best practices for running Airflow in production.
AI-written summary · about
chapters (11)
- 0:00Welcome
- 1:06Late data problem
- 4:59Manual limits and orchestration
- 8:49Apache Airflow
- 11:17Airflow DAGs and operators
- 16:20Airflow UI demo
- 21:25Airflow practical case
- 29:15Airflow best practices
- 33:29Managed Airflow in cloud
- 37:49Airflow on Google Cloud
- 41:22Summary
related talks
35mBecome a Conductor with Apache Airflow
16mBridging Data Science and Data Infrastructure with Apache Airflow
01h 08mThe State of Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
22mOrchestrate your data pipelines using Apache Airflow
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
36mElegant data pipelining with Apache Airflow
56mWhy Airflow is a secret ingredient in MLOps