Streamlining AI Workflows with Apache Airflow and Ray
summary
Shows a runnable Airflow + Ray + Anyscale pipeline that continuously ingests Twitch and Discord feedback, stores it to S3, then Mistral 7B fine-tuned with LoRA using Ray on Anyscale via a submit_anyscale_job operator. Uses Airflow data-driven scheduling via Airflow datasets and dynamic task mapping, counts records to gate fine-tuning, and runs automated champion–challenger evaluation with a rollout_anyscale_service operator. Providers are open source, deferred submission avoids blocking workers, and the pipeline demonstrably completes retrain → evaluate → promote or rollback.
AI-written summary · about
chapters (14)
- 0:00Introduction to Airflow and Ray
- 1:48Airflow and Ray ecosystems
- 2:40MLOps lifecycle with Ray
- 3:13Anyscale platform overview
- 3:30Project architecture
- 3:58Continuous ML workflows
- 4:15Fine-tuning implementation
- 5:55ETL and data-driven dags
- 8:00Challenger model deployment
- 9:54Automated retraining logic
- 11:32Evaluating challenger models
- 12:28Ray and Anyscale integration
- 14:31Key takeaways and resources
- 16:01Q&A session
related talks
29mScaling AI workloads with Ray & Airflow
25mData Science at Scale: From Notebook to Production with Airflow and Ray
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
49mOrchestrating data and ML workflows with Apache Airflow
27m[Opening Keynote] Anyscale Demo: Machine Learning Application from Dev to Prod
17mScaling and Unifying SciKit Learn and Spark Pipelines using Ray