Latency SLOs Done Right

on YouTube ↗27m

" summary

Claim: percentiles alone mislead when aggregating across nodes and long windows, producing huge errors (example: ~68% error from naive averaging). Practical alternatives are three methods: keep raw logs (accurate but costly), use counter metrics that count requests below a chosen threshold (cheap, requires fixed thresholds), or store HDR histograms/DDSketch-style summaries for efficient, aggregatable distributions. Benchmarks show sketches and HDR implementations achieve ~0.01–0.25% accuracy and performant aggregation; Circonus’s histogram tooling demonstrates this works in production.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 4:40What is an SLO?
3. 13:34Latency SLOs via Log Data
4. 15:13Latency SLOs with Counter Metrics
5. 16:16(2) Latency SLOs via Counter Metrics
6. 18:38HDR Histograms
7. 19:40HDR Histogram Data Structures
8. 22:01HDR Histogram Metric
9. 22:16Latency SLOs with Histograms

" related talks

• 31mLatency SLOs Done RightFred Moyer · SREcon19 Americas
• 32mSoftware Performance: A Shape Not a NumberKay Ousterhout · Strange Loop 2018
• 36mHow to Include Latency in SLO-based AlertingBjörn Rabenstein · KubeCon + CloudNativeCon North America 2019
• 30mTechniques for SLOs and Error Budgets at ScaleFred Moyer · Conf42 Observability 2023
• 39mThe Many Ways Your Monitoring Is Lying to YouSebastian Kirsch · SREcon16 Europe
• 37mLies, Damn Lies, and MetricsAndre Arko · Strange Loop 2016
• 38mTechniques and Tools for a Coherent Discussion about Performance in Complex...Theo Schlossnagle · SREcon16 Europe
• 43mFinding the Golden Signals with PrometheusJack Neely · All Things Open 2020