Lightflow - A lightweight, distributed workflow system
summary
Lightflow is a lightweight Python DAG workflow system built to run per-file pipelines at the Australian Synchrotron, using NetworkX, Celery, Redis, MongoDB and cloudpickle to serialize payloads. It spawns and composes small graphs at runtime (dynamic sub-DAG spawning), runs workers without a central daemon and scales horizontally; deployed at a micro‑crystallography beamline to squash many detector files into archives. It passes modest payloads (tens to low hundreds of MB over the network), is not suited to multi‑GB 3D arrays, and does not guarantee safe partial re‑runs.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:52About me
- 1:43Australian Synchrotron
- 3:25Selected science examples
- 4:44Near realtime feedback
- 6:14Wishlist
- 8:37We had a crack at it
- 10:20Workflow schematic
- 11:04System Architecture
- 12:12Sequence: Increase a number
- 13:21Python Task - all you need
- 14:39Increase a number task logic
- 15:29Parallel tasks with lists
- 17:07Central data store
- 19:01Compress files into squashfs
- 20:12What Lightflow is good at
- 20:50What Lightflow is NOT good at
- 21:27Wait, there is more
- 22:54Source Code
related talks
38mDistributed Workflows with Flowy
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
29mDagster: a Framework for Data Processing Applications
22mAirflow Beyond the Cloud: Python Workflows at the Edge
16mExoFlow: A Universal Workflow System for Exactly-Once DAGs
01h 00mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer?
19mFlow-based programming made easy with PyF 2.0
24mLightning Talks