Come Hell or Still Water: Alleviating Tail Latency in Cloud Block Store
summary
Per-proxy dual-bucket token buckets add a main bucket (refilled every 10 ms) and a check bucket that only steady segments can draw from; hot segments drain the main bucket and naturally back off. Check-bucket size is computed from an M/M/1 queuing model with ~300 MB/s headroom. A lightweight scheduler splits the event loop into a high-priority IO queue and a low-priority queue with LT = 1.22×IO time (floor 10 μs). Results: up to 97% tail reduction in bursts, 43% underloaded, deployed across dozens of clusters serving hundreds of trillions of I/Os.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:23Cloud block storage architecture
- 3:53Burst events impact
- 6:44Failed latency mitigation
- 9:36Key constraints and ideas
- 10:40Dual-bucket throttling
- 12:50IO task prioritization
- 14:17Evaluation with production traces
- 16:23Production deployment results
- 18:10Summary and conclusions
related talks
52mMeasuring and Optimizing Tail Latency
28mAchieving Rapid Response Times in Large Online Services
19mFlash Sale Engineering
21mStateless Datacenter Load-balancing with Beamer
45mEverything Will Flow
42mPredictive Load-Balancing: Unfair but Faster & more Robust
19mBreaking Storage Barriers: How RabbitMQ Streams Scale Beyond Local Disk
10mConsiderations for a large-scale low-latency system