Finding the Golden Signals with Prometheus
summary
Prometheus works but its histogram percentiles are often inaccurate — errors of 200–300% on tail estimates — so the talk shows concrete workarounds: build a templated kit of common metrics, emit standard labels, record SLI/SLOs as three counters instead of histograms (total, errors, over‑threshold), and push recording rules into Grafana/Prometheus. Example SLO: 95% over 30 days with a 5% error budget; alert on burn rate (example multiplier 14.4) so alerts clear quickly and catch problems early.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:44WHAT IS PROMETHEUS?
- 5:17USE PROMETHEUS EFFECTIVELY
- 6:18GOAL OF SITE WIDE INSTRUMENTATION
- 10:51METHODS CREATE SERVICE LEVEL INDICATORS
- 12:38AN IDEAL WORKFLOW: SRE ABSTRACTION LAYER
- 15:09ARTICULATING A SERVICE LEVEL OBJECTIVE
- 17:32PROMQL: AVAILABILITY RATIO SLO EXAMPLE
- 18:54PROMQL: LATENCY SLO EXAMPLE
- 20:34HISTOGRAMS AS A SKETCH FOR PERCENTILE ESTIMATION
- 27:12ALERTING TERMINOLOGY
- 28:16ERROR BUDGETS
- 28:49CALCULATING ERROR BURN RATES
- 30:11PROMQL: BURN RATE EXAMPLE
- 32:09TAKEWAY POINTS
- 33:58REFERENCES AND LINKS
related talks
50mApplication metrics with Prometheus
37mPractical Capacity Planning using Prometheus
36mHow to Include Latency in SLO-based Alerting
31mEvolution of Metric Monitoring and Alerting: Upgrade Your Prometheus Today
22mLook Ma, No Metrics: A Quest Towards Extracting SLOs
46mCluster Golden Signals to Avoid Alert Fatigue at Scale
33mAlerting in Cloud Native Environments
23mGrafana SLO Demo: Prioritize critical resources with SLO-driven IRM