Service Degradation at Scale: Creating Instant Capacity
summary
Service degradation disables noncritical features to reclaim compute instantly, with concrete targets of 5/10/20% CPU savings for Levels 3/2/1 and validated via weekly A/B and quarterly full‑scale tests (100% production traffic) to measure system utilization and engagement impacts. A centralized "unified site degradation" orchestration picks preapproved policies to slow demand growth (used during the Oct 4 2021 outage) and lets the fleet handle multi‑data‑center failures by combining ~10% spare capacity with automated degradation to survive two‑region outages.
AI-written summary · about
chapters (9)
- 0:00Service degradation intro
- 1:07Why service degradation program
- 4:56History and core values
- 8:08Defcon levels and testing
- 12:39Scaling degradation across services
- 17:32Lessons from Oct 4 2021 outage
- 21:48Unified site degradation plans
- 25:00Multi-region failure handling
- 26:02Key takeaways
related talks
46mChaos Experiments - Datacenter Stress Testing
43mGoogle - Load-shedding
25mBuilding a highly scalable system to assure Business Resilience
01h 08mPerformance testing in the cloud
22mScaling Culture of Resiliency in the Enterprise at Charter Communications
45mPerformance-by-Design: Embedding Intelligent Scaling and Guardrails Into Platform Engineering
15mDevOps at scale
02mOpening Remarks