Performance Testing of Prometheus Based Metric Platforms
summary
Hootsuite builds a Locust-based test framework (a custom Locust with aiohttp client and a dashboard JSON generator) that replays Grafana behavior by serially loading template variables then concurrently requesting all panel queries. Tests run for 1/5/10 virtual users over 1/3/7/30‑day windows, each combination repeated 10 times. Initial 30‑day queries show huge latency and error rates; comparing AMP and PromScale confirms the problem. Seventeen tuning iterations on Thanos (more CPU/memory, hash partitioning, memcached move, cache and concurrency tweaks) cut error rates from ~21% to ~1%.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:44Background
- 3:30Model
- 8:18Methodology
- 11:32Testing Platform
- 19:50Testing Data
- 21:49Identifying Current Performance
- 24:40Benchmarking Changes
- 28:45QA
- 37:02Outro
related talks
50mApplication metrics with Prometheus
09mFun and Games: Using Prometheus to aggregate disparate metric dat...
37mCloud-native Metric Monitoring with Prometheus
27mObservability with Prometheus and beyond
31mHandling Billions of Metrics with Prometheus and Thanos
26mWhen Prometheus Can’t Take the Load Anymore
10mUpstream-First, High Scale Prometheus Ecosystem
15mScaling Observability Stack for AIOPS