Orchestrating data and ML workflows with Apache Airflow
summary
Pipeline built in Apache Airflow with eight DAGs and dataset-driven scheduling, using S3 for images, DocDB for metadata, Hugging Face ResNet50 fine-tuned via a custom operator, and Slack alerts. Uses async S3 key sensor to free worker slots, dynamic task mapping to handle hundreds of files, and the Astro SDK for tool-agnostic tasks. Baseline model performs near guessing; after fine-tuning the model reaches AUC 0.844. Deployment task is scaffolded but not pushed to production.
AI-written summary · about
chapters (34)
- 0:00intro
- 0:22preface
- 1:53ml orchestration ∈ [ml ops]
- 3:21automatable components
- 4:47airflow crash course
- 5:59airflow ui
- 6:58dags - tasks - operators
- 10:02dags complex as you want
- 10:26why airflow?
- 13:01the data
- 14:20sometimes it is (relatively) easy
- 15:37sometimes it is harder
- 15:53the pipeline
- 16:12the tools
- 17:118 dags, 6 datasets
- 19:59continuous
- 20:39two dags waiting for new train/test data
- 21:18deferrable operators can save resources!
- 23:30dynamic tasks
- 25:002 dags handling preprocessing
- 27:04astro sdk - part 1
- 28:36train the model
- 29:13wrapping model fine-tuning into a custom operator
- 30:37get a baseline
- 31:11wrapping model testing into a custom operator
- 31:35test fine-tuned model
- 32:02airflow notifiers
- 33:29customized slack alerts
- 34:11deploy the best model - astro sdk part 2
- 36:08demo
- 42:12the results
- 43:17what is next?
- 44:56airflow ♥ ml - resources
- 48:38thank you
related talks
16mBridging Data Science and Data Infrastructure with Apache Airflow
14mScheduling machine learning pipelines using Apache Airflow
21mStreamlining AI Workflows with Apache Airflow and Ray
24mAutomating ML Pipelines With Kubernetes And Airflow
32mMaking Happy Modelers: Build and Maintain...
18mAutomate the boring ML stuff with pipelines
33mDeploying and Scaling Spark ML and Tensorflow AI Models
34mData Products, From 0 to data science pipeline