Scheduling machine learning pipelines using Apache Airflow

on YouTube ↗14m

" summary

Shows how to schedule ML batch inference pipelines with Apache Airflow using per-attendee Airflow on AWS and Jupyter notebooks for DAGs. Demonstrates programmatic Python DAGs, managed connections and secrets, the scheduler handing out tasks to parallel workers and web UI monitoring, and how to extend operators. Provides hands-on exercises, a live instance per participant that is torn down after about five minutes, and concrete examples of ETL, model inference and orchestration.

AI-written summary · about

" chapters (6)

1. 0:00Apache Airflow for ML pipelines
2. 3:38Apache Airflow core features
3. 6:30Airflow vs alternatives
4. 8:41Airflow architecture
5. 11:34Workshop setup
6. 13:20Workshop resources

" related talks

• 16mBridging Data Science and Data Infrastructure with Apache AirflowDaniel Imberman · PyCon US 2020
• 49mOrchestrating data and ML workflows with Apache AirflowTamara Janina Fingerlin · Conf42 Machine Learning 2023
• 46mGetting Started With Data Pipelines Using Apache AirflowRicardo Sueiras · Big Data Conference Europe 2022
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 36mElegant data pipelining with Apache AirflowBolke de Bruin · PyData Amsterdam 2018
• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 42mHow I learned to time travel, or, data pipelining and scheduling with AirflowLaura Lorenz · PyData DC 2016
• 24mAutomating ML Pipelines With Kubernetes And AirflowYana Segal · DataEngBytes 2020