Distributed Workflows with Flowy
summary
A library called Flowy models distributed asynchronous workflows as regular single‑threaded Python code and infers a dynamic DAG by replacing calls with nonblocking proxies; a local backend demo runs the example in about 2 seconds versus ~10s sequentially. Execution works with backends like Amazon’s SWF, supports heartbeats, retries and timeouts, uses subworkflows to keep history small, requires idempotent tasks, and scales to long‑running, multi‑machine deployments.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:07Video Processing Workflow
- 10:30Workflow Engines
- 19:22Execution Path
- 20:06Special Exceptions
- 26:57Scaling
- 32:22Comparison
related talks
30mLightflow - A lightweight, distributed workflow system
19mFlow-based programming made easy with PyF 2.0
19mDistributed Workflows for Microservices-Style Applications [I]
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
59mHustle and Flow
36mGetting Started with Apache Airflow - Building Your First Workflow
39mWorkflows, a new abstraction for distributed systems
24mBuilding Workflows With Celery