Latency SLOs Done Right
summary
Slides present three concrete methods: parsing logs (accurate but costly — about 100 bytes per request → 1 GB per 10M requests, often stored in Splunk/Elasticsearch), counting requests by threshold (simple counters, efficient, but threshold is fixed and not retroactive), and histogram-based quantiles (Circonus log‑linear and HDR histograms, mergeable and volume‑invariant, ~300 bytes in memory, C implementation with Python bindings, ns inserts). Example error: averaging p95s 220ms and 650ms gives 430ms while true combined p95 is 230ms — a ~200% mistake; worst‑case bin error ≈4.3%.
AI-written summary · about
chapters (14)
- 0:00Introduction to latency
- 2:04Speaker background
- 2:45Defining SLOs and SLIs
- 4:04SLA vs SLO
- 6:32Mistakes in calculating SLOs
- 10:45Tools for log analysis
- 11:53Calculating with log data
- 15:00Counting requests
- 17:35Understanding histograms
- 20:01Log-linear histograms
- 22:58Calculating SLO with histograms
- 25:47Pros and cons of histograms
- 27:21Python implementation
- 28:56Conclusion and resources
related talks
27mLatency SLOs Done Right
36mHow to Include Latency in SLO-based Alerting
32mSoftware Performance: A Shape Not a Number
30mTechniques for SLOs and Error Budgets at Scale
22mLook Ma, No Metrics: A Quest Towards Extracting SLOs
39mFast log management for your infrastructure
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
23mGoing from 30 to 30 Million SLOs