Techniques for SLOs and Error Budgets at Scale
summary
Rolled SLOs across a 1,000+ engineer org by prescribing concrete SLIs (good vs bad requests), calibrating thresholds until 99.95% (3.5 nines) is achieved, and exposing dashboards that let teams iterate latency cutoffs. Measures use raw histograms (log‑linear or inverse cumulative histogram) to sum exact sample counts and avoid sketch/T‑digest aggregation errors. Error budgets are literal: success 99.95 gives error budget = 0.05%, and composite budgets roll upstream through third‑party and mid‑tier services to prioritize reliability work.
AI-written summary · about
chapters (19)
- 0:00intro
- 0:21preface
- 0:54have you used this in your career? (traffic for total.rrd)
- 1:16hi, i'm fred
- 2:00how do you implement slos for 1000 engineers?
- 2:49books
- 4:33sli: good vs bad requests
- 7:11slo: good/bad time_range
- 10:42eb: 1-slo, 1-0.9995 = 0.05%
- 12:33keys to slo / error budget democratization
- 13:18latency and availability
- 15:05measuring availability is easy, measuring latency is not easy
- 16:17quantifying latency at scale
- 17:22a common mistake
- 19:26"dr. histogram - how i learned to stop worrying and love latency bands"
- 24:01use raw histograms, avoid sketches & approximations
- 25:57decomposing histogram modes
- 27:54multi service slos / error budgets
- 29:57thank you, questions?
related talks
19mLatency and Availability Error Budgets Done Right at Scale
30mMetrics for good, not evil
30mReliable Observability at Scale: Error Budgets for 1,000+
38mImplementing SRE practices: SLI/SLO deep dive
17mRolling out Service Levels across a large org
09mHow to answer tricky questions the SRE way
23mGoing from 30 to 30 Million SLOs
29mMeasuring Reliability: What Got Us Here Won't Get Us There