High availability by offloading work
summary
Shopify handles tens of thousands of jobs per second and peaks from BFCM 2020 hit over 5 billion USD in sales with peak rates above 100 million USD per hour while traffic spikes reach ~170k requests/s. The stack uses Redis-backed queues and RabbitMQ plus Kafka event logs, implements Sagas and command-orchestrator patterns, checkpointed interruptible jobs, and a job-iteration framework on GitHub. Engineering achieves high availability despite async guarantees by versioning, at-least-once strategies and deploying about 40 times a day, proven by a Kafka-powered real-time visualization.
AI-written summary · about
chapters (5)
- 0:00Start
- 0:53Introduction about High Availability and Offloading Work
- 2:37Background Jobs
- 12:08Events Choreography
- 28:52Message Oriented Middleware
related talks
33mShopify in Multiple Datacenters
46mBlack Friday Brilliance: Managing a Billion Transactions with Tech, Tactics
47mApplying “web scale” patterns (...)
22mBlack Friday Brilliance: Managing a Billion Transactions with Tech, and Teamwork
45mSelling Food With Elixir
01h 02mAsynchronous Distributed Systems
37mWhy it is harder to scale with stateful, data-driven systems
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)