Elegant data pipelining with Apache Airflow

on YouTube ↗36m

" summary

Claims that pipelines must be idempotent and deterministic and use immutable partitions guide the concrete examples: Airflow templates execution_date, defines inlets and outlets with versioned S3 outputs, and can wire Spark model runs to load predictions into Apache Druid so reprocessing yields identical partitions. Airflow shows lineage to Apache Atlas, supports Kubernetes executor task dependencies in 1.10, and is already used by 120+ companies with 60+ contributors while visualizers can struggle at thousands of tasks.

AI-written summary · about

" chapters (12)

1. 0:00Apache Airflow intro
2. 1:28What is Airflow
3. 4:22Directed acyclic graphs in Airflow workflows
4. 6:51Airflow for pipelines
5. 7:57Elegant pipelines
6. 11:10Functional programming
7. 14:45Idempotency in processing
8. 18:22Lineage in transformations
9. 23:38Idempotent pipeline
10. 27:58Currency prediction demo
11. 29:22Conclusion and key takeaways
12. 30:38Q&A

" related talks

• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 32mOrchestrating Your Data Pipelines with Apache AirflowRicardo Sueiras · GOTO Amsterdam 2022
• 46mGetting Started With Data Pipelines Using Apache AirflowRicardo Sueiras · Big Data Conference Europe 2022
• 48mFlow Is in the Air: Best Practices with Apache AirflowDominik Benz · PyCon.DE 2017
• 38mUse Apache-Airflow to Build Data WorkflowsLuther Hill · PyData Indy 2019
• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 16mBridging Data Science and Data Infrastructure with Apache AirflowDaniel Imberman · PyCon US 2020
• 42mHow I learned to time travel, or, data pipelining and scheduling with AirflowLaura Lorenz · PyData DC 2016