Managing incidents in microservices
summary
Argues that incident work must be driven by measurable SLOs, using 99.9% of HTTP calls under 100ms as an example and an explicit error budget to decide paging. Recommends putting developers on call or using SREs depending on maturity, cites two‑pizza teams, Google SRE and Amazon models, and four incident roles: incident commander, communications officer, scribe, subject matter experts. Advocates distributed tracing with tools like AWS X‑Ray, Jaeger/OpenTracing and SLO‑based alerting (e.g. alert if error rate >0.5% for 10 minutes), plus suppression, grouping and follow‑the‑sun schedules.
AI-written summary · about
chapters (10)
- 0:00Start
- 6:30A good Service Level Object
- 10:55Put developers on-call
- 13:41Is there an industry standard?
- 17:23Incident Commander Responsible for managing the incident response process and providing direction to the responder teams.
- 21:04Scribe Responsible for documenting information
- 21:53Subject Matter Expert
- 31:00Severity levels to the rescue
- 33:14Scaling alerts in microservices
- 36:06Recommended Reads
related talks
30mHow Did Things Go Right? Learning More from Incidents
15mSustainable Incident Management for happy SRE teams
27mGetting Started with SRE
31mFuturistic Luxury Incident Response
15mMaintaining reliable systems
47mPersistent SRE Antipatterns: Pitfalls on the Road to Creating a Successful
06mIGNITE: Want to start with SRE? Start with this talk.
19mHow to handle incidents at scale