Defcon: Preventing Overload with Graceful Feature Degradation
summary
Defcon is a system that lets operators disable noncritical features to prevent overload by trading user impact for capacity savings, using Defcon knob actuator to coordinate thousands of knobs and automated A/B and full-scale production tests to measure resource savings. Meta measures up to ~40% unexpected load spikes in one pandemic example, classifies degradations as Defcon levels 1–4, runs weekly and quarterly validation, and successfully averts a cascading site outage by rolling to level 3 then 2.
AI-written summary · about
chapters (9)
- 0:00Intro
- 3:54Key Idea
- 6:30Defcon Levels
- 7:59Challenges
- 8:42Automated Testing
- 9:52Resource Savings
- 11:33Knob Actuator Service
- 12:06Real World Example
- 13:50Lessons Learned
related talks
20mHelp Protect Your Data Centers with Safety Constraints
37mReal-Time Adaptive Controls for Resilient Distributed Systems
13mTen Ways to Doom Your DevSecOps
56mDefending against CDD: Chaos-Driven Delivery
47mLoad Shedding—Approaches, Principles, Experiences, and Impact in Service Management
01h 04mPredictably Unpredictable
31mDefence at the Boundary of Acceptable Performance
01h 00mConfiguration, Extension, Maintainability