How We Reliably Send a Million Messages Per Request
summary
A bulk messaging system moves from a queue-of-microservices to Temporal workflows, then hits namespace action limits at ~55,000 requests in a minute (≈900 RPS) with 22,000 failures; fix is to add front-end backpressure plus architectural changes: eliminate per-message child workflows, replace with a single long-running, heartbeating batch activity (progressive batching) to make actions O(runtime), then shard that work across N parallel streaming activities with selectors (sliding window), use continuous new to truncate history, and serialize shared-entity updates via signal-driven workflows.
AI-written summary · about
chapters (17)
- 0:00Introduction and speaker welcome
- 1:51What is Twilio and the problem
- 3:58Initial queue-based design issues
- 5:23Resource exhaustion incident at Tahoe
- 7:48Adding back pressure to Temporal
- 8:53First redesign with multiple activities
- 10:37Killing per-message child workflows
- 12:00Progressive batching in one activity
- 14:31Limitations of single-activity approach
- 15:35Introducing sliding window pattern
- 17:18Full architecture with sliding window
- 20:31Shared state and race conditions
- 21:59One workflow per entity with signals
- 23:01Key lessons and takeaways
- 24:04Why use Temporal instead of building yourself
- 26:32Three hard-learned operational tips
- 27:35Team and closing remarks
related talks
43mRealtime performance at scale (...)
33mConversational Big Data
54mA million WebSocket connections and pub/sub
39mTop Performance Challenges in Distributed Architectures
41mPushing RabbitMQ to the limit
40mThe Road to 2 Million Websocket Connections in Phoenix
38mPerformance Engineering At MasterCard
25mGame time: A playbook to (unsuccessfully) 10x in a week and (successfully) 10x in a year