Diving through RCAs a quest for reliability
summary
Postmortem analysis at Elastic finds top root causes are upgrades, network changes and security, and one live kernel patch causes a region outage impacting 1,000+ customer deployments and ~750 internal systems with a ~5 hour outage and ~7 hour impact window. Response shifts from StatusCake to Kibana alerting, adds synthetic uptime checks, SLOs and burn‑rate alerts, integrates an observability AI assistant trained on docs/runbooks/GitHub, halves incidents via infra cleanup and moves workloads toward serverless/Kubernetes.
AI-written summary · about
chapters (20)
- 0:00Root cause analysis intro
- 1:15On-call stress pressure
- 2:39Root cause analysis process
- 4:26Post-mortem culture focus
- 5:06Top incident causes at Elastic
- 7:12Linux kernel patch example
- 9:39Challenges in root causes
- 10:25Five Ws method for RCA
- 11:08Reactive vs proactive management
- 11:50Improving observability
- 13:35Elastic observability stack
- 14:39Demo SLOs and OpenTelemetry
- 16:50Elastic AI troubleshooting
- 17:11Infrastructure improvements
- 17:56Documentation and feedback
- 18:40Automating incident platforms
- 19:46System design changes
- 20:51OpenTelemetry contributions
- 21:33Human factor in response
- 22:39Conclusion and resources
related talks
40mHow to SRE When Everything's Already on Fire
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
45mEngineering resiliency - how I learned to love an outage
25mLive Q&A Session #2
30mThere is No Root Cause...
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...
29mCulturing Resiliency with Data: A Taxonomy of Outages
31mEscaping operations hell