Stop Rate Limiting! Capacity Management Done Right
summary
Argument: rate limits mislead because throughput, latency and concurrency are different variables; using Little's Law the talk shows concrete failure modes (origin with seven worker threads and 2s response time only drains 3.5 req/s so a 5 rps client produces an unbounded queue and timeouts). Solution: enforce concurrency control (limit in‑flight requests) plus an adaptive probe (implemented in NGINX+Lua) that uses additive increase multiplicative decrease to estimate capacity, borrow unused quota by percentage, and stabilizes throughput in the demos.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:05API Management Gateway
- 1:50Why Rate Limiting Doesnt Work
- 2:28Queueing Theory
- 4:45Demos
- 11:19Queue Theory
- 18:47Limit Concurrency
- 20:14throughput vs capacity
- 21:54TCP congestion avoidance
- 23:22Adaptive capacity management
- 28:17How capacity gets managed
- 31:36When to admit a request
- 37:04When to borrow quota
- 40:19Summary
related talks
38mA Practical Look at Performance Theory
26mTraffic Management w. Rate Limiting to Ensure API Scalability & Resiliency
06mPractical performance theory
16mLightning Talk: More Intelligent Global Rate Limiting
44mDeep Dive talk | Resilience in Microservices with Congestion Control
30mQueueing Theory, In Practice: Performance Modelling in Cloud-Native Territory [I]
30mWhat's the Cost of a Millisecond?
19mFlash Sale Engineering