Creating extensible workflows with off-label use of Python
summary
Claims that Python type annotations drive static rule graphs, that async/await coroutines yield runtime dependencies via get and multi‑get for parallelism, and that frozen dataclasses provide stable fingerprints so caching and remote execution work. The runtime event loop lives in a Rust engine while task logic stays in Python, and this combination powers the Pants workflow API, delivering fine‑grained tasks, concurrency, reproducible caching, and easy extensibility.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:51About me
- 1:21What is a workflow?
- 2:16Workflows show up all over the place
- 2:31Example: Processing Uploaded Images
- 4:27Workflow is defined by a task graph
- 5:25Workflow system design
- 6:46Workflow for software builds
- 9:08Design goals
- 11:16Task API design challenges
- 12:05Python to the rescue!
- 14:25Building the rule graph
- 15:04Static validation
- 17:07Type annotations provide
- 18:20Why coroutines?
- 19:58Coroutines are powerful
- 20:39Coroutines can express concurrency
- 21:22Coroutines help avoid side effects
- 22:49Coroutines provide natural control points for
- 24:41Summary
related talks
44mImplementing a Workflow Engine in Python
51mTaking Python beyond scripting
36mPumping up Python modules using Rust
44mHow Python Harnesses Rust through PyO3
31mPumping up Python modules using Rust
44mDo Pythons Rust? How we used PyO3 to build a Python Stream Processor w/ a Rust Heart
33mBuilding with Multiple Languages: A Python-First Approach
38mDistributed Workflows with Flowy