Google - Load-shedding
summary
After three global outages in 2009 engineers build and deploy a load‑shedding stack by 2010 that evolves over seven years and actually prevents cascading failures while raising utilization from typical ~20% to ~80–90% on packed machines. Specifics include sustained peak performance via server‑side max‑in‑flight and token buckets, cost per second accounting instead of QPS, priority bands (criticality), micro‑failover with gRPC channel pickers, and sidecar proxies like Envoy and Linkerd and higher‑level control with istio, all of which work in production.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:56Google detailed application view
- 5:11Load Shedding: Threats
- 9:01Capacity during outages
- 11:34Load Shedding: Principles
- 12:12Principles: Sustain peak performance (1/6)
- 13:01Principles: Isolation (2/6)
- 14:09Principles: Criticality (3/6)
- 15:00Outage behaviour
- 16:08Principles: Work conserving (4/6)
- 17:21Principles: Cost-based (5/6)
- 18:53Principles: Retries (6/6)
- 22:07Traffic control points
- 22:46Proxying variations server side wrapper
- 23:07Client-side techniques
- 26:14Client-side-rate limiting
- 27:06Client-side Circuit breaker pattern
- 28:00Server-side stack
- 28:59Server side techniques
- 32:10Load task Thresholds
- 35:03Rate limit implementations
- 35:25Token-bucket based rate-limiting
- 35:54Deployment wrappers: Envoy
- 36:51Deployment wrappers Linkerd
- 37:13Istia - Service management platform
- 38:00Use case 1/3. Reliability and Utilization
- 39:44Normal traffic pattern
- 40:19Combined provisioning
- 41:26Use case 3/3. Product-level & service-level decisions
related talks
39mOops. I broke the Google, now what?
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
52mPerformance & Stability Anti patterns
27mService Degradation at Scale: Creating Instant Capacity
16mRunning Services at Google Scale
18mScaling APIs to Billions: Extreme-Scale Reliability
20mUber's Failover Architecture: Reconciling Reliability and Efficiency in Hyperscale...
32mLoad Shedding for Evil Wizards