Bridging Data Science and Data Infrastructure with Apache Airflow

on YouTube ↗16m

" summary

Shows how Apache Airflow moves a prototype from a single-machine Jupyter Notebook to a distributed pipeline running Spark + TensorFlow, using custom Airflow operators to provision resources, launch training, run hyper-parameter tuning, and perform canary tested deployment to Google Cloud Functions. Demonstrates the exact operators, resource configs (CPU/memory and network needs), orchestration DAGs, and monitoring hooks used, and reports what failed, what scaled, and what latency and cost looked like.

AI-written summary · about

" chapters (5)

1. 0:00Introduction to Airflow
2. 3:38Data pipeline challenges
3. 7:40Airflow for data orchestration
4. 12:19Real-world use cases
5. 15:03Q&A and closing

" related talks

• 46mGetting Started With Data Pipelines Using Apache AirflowRicardo Sueiras · Big Data Conference Europe 2022
• 14mScheduling machine learning pipelines using Apache AirflowAxel Goblet · PyData Eindhoven 2019
• 01h 59mIntroduction to orchestration ETL processes with Apache AirflowMiroslav Tomic · DSC Europe 23
• 34mData Products, From 0 to data science pipelineDaniel Imberman, Gonzalo Diaz · PyData Córdoba Argentina 2019
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 36mElegant data pipelining with Apache AirflowBolke de Bruin · PyData Amsterdam 2018
• 49mOrchestrating data and ML workflows with Apache AirflowTamara Janina Fingerlin · Conf42 Machine Learning 2023
• 17mGoing with the flow? Apache Airflow for operational-quality scientific workflowsPatrick Sunter, Michael Pegios, Daehyok Shin · PyCon AU 2025