How Affirm Uses Temporal as a Batch Data Plane
summary
Affirm replaces fragile Kubernetes pod-operator runs inside Airflow with Temporal as a batch data plane, using an ephemeral worker controller that scales user workers scale-from-zero to match Spark jobs (20 minutes to hours), proxies requests, monitors health, and scales down. The system uses unique workflow IDs and client reattachment for safe retries, returns structured diagnostics that separate infra vs application errors with stack traces and runbooks, and standardizes execution across batch jobs — deployed and improving platform visibility and reliability.
AI-written summary · about
chapters (7)
- 0:00Introduction and speakers
- 1:06Existing Airflow batch platform
- 3:17Problems with Kubernetes pod operator
- 6:07Why choose Temporal
- 7:30Ephemeral worker controller design
- 9:57Benefits and improvements
- 10:40Key takeaways and future plans
related talks
26mTemporal: A Better Way to Build Modern Applications
26mRethinking Cloud-Native Patterns on Kubernetes
23mDesigning a Workflow Engine From First Principles
26mMicroservices Madness: Application And Infrastructure Automation Using Temporal
30mOrchestrate Your Business Workflows with Temporal
42mUsing Temporal Workflows for Seamless Dev + Research Collaboration & Continuous Algo D
26mOperating Cassandra in Kubernetes with Temporal
33mFrom Prototype to Production: Making Agentic AI Systems Reliable