From Nothing to Everything: One Single Alert to Managing Datacenters
summary
Starts from a single Temporal workflow that turns a “GPU fell off PCIe bus” alert into a JIRA ticket and grows into one persistent workflow per server that supervises racking, provisioning, maintenance and return to service. Uses Temporal, OpenTelemetry, child workflows, signals and Protobufs; initial critical workflow is ~200 lines of logic; system runs as a single process for over a year with <1GB memory per worker. Entity pattern solves races, child workflows must surface human-readable failure headlines, and workflow history becomes a training corpus for AI decisions.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:30Crusoe and GPU challenges
- 3:14First Temporal workflow demo
- 5:18Why Temporal for alerts
- 6:22Expanding workflows to more issues
- 8:26Modeling machines as workflows
- 10:11Child workflow architecture
- 11:58Scaling and lifecycle benefits
- 13:23Debugging large-scale workflows
- 14:46AI integration with Temporal
- 16:28Platform engineering lessons
- 17:30Protobuf and upgrades
- 18:58ROI and business impact
- 20:03Evolution of the system
- 21:47Final insights and Q&A
related talks
23mDesigning a Workflow Engine From First Principles
30mOrchestrate Your Business Workflows with Temporal
26mMicroservices Madness: Application And Infrastructure Automation Using Temporal
50mThe end of event-based architectures? Orchestration with Temporal
42mHow to build distributed systems in JavaScript
42mUsing Temporal Workflows for Seamless Dev + Research Collaboration & Continuous Algo D
01h 38mBuilding Reliable Distributed Systems
33mFrom Prototype to Production: Making Agentic AI Systems Reliable