Doing Things Prometheus Can’t Do with Prometheus
summary
Argues that Prometheus handles most observability needs if used thoughtfully: deploy HA by running replicas behind a proxy or using Thanos/Promxy to fan-out queries; control cardinality, keep ~50% CPU/memory headroom, set a 50 million sample limit and max concurrent queries (~1 per CPU); use disk-backed retention (one deployment shows ~250 days disk-backed retention) or stitch regional shards/federation; write exporters with const metrics to avoid zombie series; do simple anomaly detection with stddev or predict_linear and compute SLOs in PromQL (example: 99th-percentile SLO, 95.8% over 24h).
AI-written summary · about
chapters (30)
- 0:00Introduction
- 0:40Clickbait
- 1:24The problem with technology
- 2:32The goal
- 3:19What are metrics
- 4:18Metrics logs and traces
- 5:32Going beyond Prometheus
- 6:40The limitations of Prometheus
- 7:42Prometheus is not a distributed system
- 9:20How to work around this
- 13:42Prometheus cardinality
- 15:08Prometheus metrics
- 15:37Prometheus limits
- 18:45Be careful with aggregation
- 19:56Crafting metrics that match desired outcomes
- 21:06Longterm storage
- 22:23New solutions
- 23:19Long term storage
- 24:09Machine learning
- 24:39Latency metrics
- 25:25Spikes
- 25:53Predict Linear
- 27:19Const Metrics
- 29:27Alert Manager
- 31:00SLOS
- 31:36Setting an objective
- 32:18Defining availability
- 32:52Prometheus subquery
- 33:52Problem QL tricks
- 34:45Conclusion
related talks
01h 19mIntro + Deep Dive: Prometheus
26mWhen Prometheus Can’t Take the Load Anymore
27mObservability with Prometheus and beyond
30mIntro: Prometheus
37mJourney inside Prometheus: Architecture, updates, outlook
36mObservability with Prometheus and Beyond: Leveraging Cloud-native Technologies Outside of Cloud
24mFrom Monitoring to Observability: 10 Things to Do with Prometheus in 2020
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack