Designing a Workflow Engine From First Principles
summary
Temporal is a workflow engine built to preserve full program state durably and run millions of bounded workflows by scaling out shards rather than scaling up single instances. It guarantees transactional updates across state, timers and task dispatch by writing tasks into transfer queues inside the same shard and then asynchronously exporting to a global queue, avoiding two‑phase commits. Production uses include over 100 Cadence/Temporal users, core storage on Cassandra, search via Elasticsearch, and multi-cluster failover for region outages.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:52Defining workflows
- 6:44Workflow engine basics
- 9:55Why transactionality is critical in workflow engines
- 12:38Horizontal scaling
- 16:29Transfer queues for transactions
- 19:16Batch operations
- 20:19Temporal cluster architecture
- 21:45High availability deployment
- 22:26Summary
related talks
26mTemporal: A Better Way to Build Modern Applications
30mOrchestrate Your Business Workflows with Temporal
33mLessons learned using Temporal to run Temporal Cloud
31mTemporal's Director of Eng and Principal Software talk High Availability in Temporal
50mThe end of event-based architectures? Orchestration with Temporal
42mUsing Temporal Workflows for Seamless Dev + Research Collaboration & Continuous Algo D
42mHow to build distributed systems in JavaScript
26mMicroservices Madness: Application And Infrastructure Automation Using Temporal