Oops! Learning From Surprise
summary
Claim: operational surprises labeled oops yield learning beyond fixes so incidents with no customer impact teach as much as severe outages. Examples: Spinnaker change floods Titus with pressure (oops 170), a Jettas Redis client bump causes connection pool leaks, and a UI/auto‑scaler interaction lets the autoscaler terminate healthy instances when users set desired size. Responses include disallowing desired when autoscaling, adding Jettas instrumentation, using a staging Spinnaker stack to recover, and leveraging demand engineering to resize many clusters. The key question is how did we get here.
AI-written summary · about
chapters (8)
- 0:00Introduction to incidents
- 3:17Beyond preventing recurrence
- 8:56Learning from others' mistakes
- 15:58Identifying operational gaps
- 19:27Investigating operational surprises
- 22:15Writing effective narratives
- 31:48Key takeaways and resources
- 33:14Q&A session
related talks
10mBuilding an 'Oops-Less Operation' at Scale
30mHow Did Things Go Right? Learning More from Incidents
01h 03mWhispers in the Chaos: Monitoring Weak Signals
02mOpening Remarks
31mThe Art of Turning Incidents into Opportunities
50mLearning in Production (or Why the Apollo 11 Landing Nearly Failed)
30mThe last in the line: a talk about DevOps and Disasters
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash