You've Lost That Process Feeling: Some Lessons from Resilience Engineering
summary
Argues that process feel—a shared, dynamic sense of where a system ought to be—is essential to handle anomalies, and that designing displays which show “where we think we should be” reduces blind AI work. Shows an incident where a tiny subset of hosts triggers panic mode and retries overload the control plane because RPC payload limits are exceeded; scaling temporarily masks the cause. Recommends tooling and practices: richer alerts, circuit breakers, throttling, quotas, production storytelling and disaster testing to enable graceful extensibility and earlier, smaller mitigations.
AI-written summary · about
chapters (9)
- 0:00Introduction to process feel
- 1:23Insights from nuclear power
- 3:29Challenges in modern SRE
- 5:11Learning from space stations
- 9:11The line of representation
- 12:33Graceful extensibility
- 16:54Saturation and interdependency
- 21:45Root cause analysis case study
- 25:56Practical takeaways
related talks
22mResilience Is Not Enough
43mResilience Engineering Mythbusting
47mPersistent SRE Antipatterns: Pitfalls on the Road to Creating a Successful
51mWithout Resilience, Nothing Else Matters
01h 05mKeynote - Without Resilience, Nothing Else Matters
05mBuilding resilience in our organizations...
05mWhat is resilience engineering
11mResilience engineering mythbusting