Scaling a Distributed Priority
summary
A horizontally scalable distributed priority queue that uses sharded MySQL with transactional writes and secondary indexes, a Fox router that keeps in‑memory distribution and priority metadata, and a thrift API to NQ/DQ/ACK/KNACK. It supports deferred delivery, 32‑bit priority, leasing with lease extensions, writer thread rate limits and continuous load testing. It handles roughly 700 billion items per day, survived a Thanksgiving 2018 write overload by adding backpressure and shard health checks, and still works while chasing thin‑spreading and global routing limits.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:44Facebook’s distributed priority queue
- 3:55Fox in async processing
- 6:02Fox with producers and consumers
- 8:32MySQL sharding
- 10:41Enqueue challenges
- 12:49Thanksgiving 2018 outage
- 13:29Fox dequeue operations
- 15:37Fox router efficiency
- 16:38Thin spreading problem
- 17:20Scaling Fox learnings
related talks
23mMaking Distributed Priority Queue Disaster Ready
46mDatabase Sharding the Right Way Easy, Reliable, and Open source
15mDistributed Caching & Queueing in the Cloud
52mTaking Storage for a Ride
32mData @Scale Seattle- Eric Hwang introduced by Ganapathy Krishnamoorthy
25mAvoiding Overload in Distributed Systems
12mCeph QoS: How to support QoS in distributed storage system
16mScale with NSQ: a realtime distributed messaging platform