Guaranteed SLAs with higher-order functions (...)
summary
Claim: a feedback control loop applied across services enforces SLAs by adjusting incoming load. Implementation uses Elixir at Grindr, a plant that measures average response time, a stateful PI controller, and a runtime drop fraction applied to requests every 100 ms loop. Setpoint is ~30 ms, system handles 3M DAU, simulations and production incidents show tail latency stays bounded and live outages are mitigated though handled-throughput can fall when overloads are extreme.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:28Backend stack
- 6:11Utilization Law illustrated (=1000, p=900)
- 8:39Control Theory
- 14:40Three queues simulated
- 16:12Three queues, naive approach
- 18:28Random dropping based on average response time
- 23:38Proportional control
- 24:37magic_number = 0.0025
- 27:44Proportional and integral controller
- 29:04Step response experiment
- 32:21Final Loop
- 32:37Simulation results 1 - steady state
- 33:43Simulation results #2 - load increase
- 36:07Real world example #2
- 40:08References
related talks
17mDefining Service Level Objectives for Exposure Notifications Servers
35mFrom Monitoring to Observability: Left Shift your SLOs with Chaos
26mTraffic Management w. Rate Limiting to Ensure API Scalability & Resiliency
37mLevel Up Your Edge: Do You Know Where Your Traffic Comes From?
37mMonitoring Service Architecture and Health with BPF
42mA blueprint for building a generic... (Ashwin Sidhalinganahalli, Fletcher Ramee)
04mLoad balancing through the years
37mHow We Try to Make a Lion Bulletproof...