Escaping operations hell
summary
Platform runs 250 microservices across two Kubernetes clusters serving 20 countries with ~80 developers and 20 external dependencies, and outages often start in networking or overloaded JVM threads. Monitoring evolves from logs to shared dashboards, synthetic transactions, alerts tuned from early warnings, and SLOs/SLIs to quantify impact; simple restarts fix ~50% of incidents. A postmortem process and shared ownership reduce repeat failures and guide prioritized fixes. 250 microservices synthetic transactions SLOs and SLIs postmortems
AI-written summary · about
chapters (14)
- 0:00Operations nightmare intro
- 2:22Signs of operations hell
- 5:09Where the system burns
- 10:23Restarts and critical fixes
- 13:51Business impact and workarounds
- 16:59External dependencies and coincidences
- 18:05Early warning systems
- 19:50Service level indicators
- 22:19Direct communication in crises
- 24:23Take responsibility for systems
- 26:08Politics during outages
- 27:53Culture of learning from failure
- 30:41Postmortems and metrics
- 31:03Stay calm and enjoy work
related talks
55mDevOps @Scale (Greek Tragedy in 3 Acts)
46mSurviving microservices
28mSurviving in a Microservices Environment
48mCloud Chaos & Microservices Mayhem
24mHow to survive Cloud – An Ops perspective
30mThe last in the line: a talk about DevOps and Disasters
24mWar Stories About the Ops in DevOps
48mDevops@Scale