Making Distributed Priority Queue Disaster Ready
summary
FOQS uses semi-synchronous MySQL replication with primaries and two secondaries across regions plus a Shard Manager integration and a routing service that implements NQ and DQ distributors to follow primaries and discover ready items. The team migrates a largest tenant processing hundreds of billions of items/day, completes 95% migrated with zero-downtime rollouts, cache-merge + randomized selection to avoid thundering herds, and achieves region-failure availability; remaining work targets sub‑second video latency and final cutover to 100%.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:31Disaster readiness for Fox
- 2:55Fox system architecture
- 6:03Regional Fox challenges
- 8:53Global Fox evolution
- 10:59Intelligent global routing
- 13:26Handling region failures
- 16:14Global item discoverability
- 19:02Fox migration challenges
- 21:10Fox future challenges
- 22:57Q&A
related talks
18mScaling a Distributed Priority
24mFailure is Not an Option: Highly Available Distributed SQL
21mBuild distributed persistent queue on FoundationDB
12mCeph QoS: How to support QoS in distributed storage system
39mZero-Downtime Rebalancing and Data Migration of a Mature Multi-Shard Platform
01h 03mClosing Keynote
54mDistributed Caching in Real-Time Payment Systems
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation