Implementing a Workflow Engine in Python
summary
Engineered a Python workflow engine that actually runs production automations for hospitals it serializes inputs with Pydantic, exposes task enqueueing via FastAPI, and uses a client-token idempotency scheme so replaying a script never duplicates side effects. Tasks go onto a Postgres-backed queue, workers persist results, and long latency human-in-the-loop steps can pause without blocking. Claims: handles complex branching workflows, avoids unsafe restarts, and already runs against Epic/Cerner integrations at large health systems.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:05Healthcare automation challenges
- 2:30Automating manual tasks
- 5:01Workflow automation
- 7:28Why Casa needed engine
- 8:51Human-in-the-loop principles
- 10:37Key requirements
- 12:43Introduction to idempotent APIs
- 15:16Idempotent operations
- 16:42AWS idempotency approach
- 19:55Replayable workflows
- 21:40Casa's engine architecture
- 23:27Async task execution
- 25:37Worker pool prioritization
- 26:00Replaying workflows
- 28:10Task input/output serialization
- 29:34Client token generation
- 33:06Workflow execution
- 36:01Q&A
- 36:44Comparison with engines
- 38:09Maintaining async order
- 39:36Open source plans
- 39:58Integration with EMRs like Epic and Cerner
- 41:44FastAPI in toolkit
- 43:09Q&A
related talks
27mCreating extensible workflows with off-label use of Python
36mHealthcare process management in Python: a use case
38mDistributed Workflows with Flowy
36mHealthcare process management in Python: a use case
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
27mModern ETL-ing with Python and Airflow (and Spark)
38mUse Apache-Airflow to Build Data Workflows
49mManaging Clinical Processes in Python: A Use Case