How to Include Latency in SLO-based Alerting
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 15:20· Room 16AB - San Diego Convention Center, Mezzanine Level·
summary
Concrete recipe: convert latency violations into errors by counting requests above a threshold (e.g., 99.9% < 1s for writes, 99.95% < 2.5s for reads) using Prometheus histograms partitioned by route/status, expose a slow-or-error ratio via recording rules, then run SLO-based alerts that combine 1-hour and 5-minute windows (long-window burn + short-window reset) with the workbook factors (e.g., 0.1% error budget, factor 14.4) so pages fire fast for outages but stop quickly after fixes.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:24What is SLO
- 1:04Read up
- 1:36SLO can go wrong
- 3:11Latency SLI is done
- 3:59Metrics vs logs
- 6:00My Alerting Philosophy
- 7:07Site Reliability Workbook
- 8:36Prometheus Talk
- 9:18Error Budget
- 11:21Error Rate
- 14:56Alert Configuration
- 15:53Alert Rules
- 17:37ISP Uptime
- 19:51Latency in SLO
- 22:31More complicated things
- 22:55Implement the easy solution
- 23:09Code X
- 25:12OS Hello
- 26:49Recording Rules
- 28:11Recording Rules Table
- 29:19Summary
- 29:31Conclusions
related talks
41mA Theory and Practice of Alerting with Service Level Objectives
43mSLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size
39mActionable Observability
31mLatency SLOs Done Right
30mTechniques for SLOs and Error Budgets at Scale
19mLatency and Availability Error Budgets Done Right at Scale
50mApplied Alerting Philosophy
43mFinding the Golden Signals with Prometheus