Airflow Beyond the Cloud: Python Workflows at the Edge
summary
Airflow 3 runs a lightweight edge executor on tiny devices like a Raspberry Pi Zero W that pulls jobs over HTTPS (no inbound ports) and mixes executors in one DAG so some tasks run in the cloud and some on the Pi. The demo builds an “On Air” LED driven by a separate service reading state files written by Airflow tasks; Zoom detection uses a macOS cpthost process instead of OAuth webhooks. It works after simulating a call but is not production-grade yet — tested on ~416 MB usable RAM with 2 GB swap added.
AI-written summary · about
chapters (13)
- 0:00Airflow at the edge intro
- 3:08Apache Airflow overview
- 4:43Airflow 3 edge executor
- 8:14Hardware setup demo
- 10:43Live demo Zoom status
- 13:21DAG operation explained
- 14:28Scheduling dependencies
- 15:57Edge executor tasks
- 17:31State files decoupling
- 19:03Design for constrained envs
- 20:19Airflow on IoT realities
- 20:47Future edge possibilities
- 21:35Resources & involvement
related talks
01h 25mApache Airflow in the Cloud: Programmatically orchestrating workloads w/ Py
36mGetting Started with Apache Airflow - Building Your First Workflow
38mUse Apache-Airflow to Build Data Workflows
01h 01mPython in Cloudflare Workers – Running Pyodide on the Edge
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
34mData Ingestion in Python using Apache Airflow
22mRunning Functions at the Edge
27mModern ETL-ing with Python and Airflow (and Spark)