Prometheus That Scales
summary
Outbrain moves from 8,000 physical servers scraping 300 million metrics per minute into Kubernetes, shards Prometheus by target to keep pods small, and layers Thanos with local MinIO on Ceph/Rook plus remote GCP. The build adds two Thanos query tiers, compactor/downsampling, and replication; it cuts S3 API traffic and saves 90% cost reduction on long-term storage while improving query latency and keeping one-year retention. Tools used: Prometheus, Thanos, MinIO, Ceph/Rook, Helm.
AI-written summary · about
chapters (15)
- 0:00Prometheus and Thanos intro
- 1:05Outbrain’s Prometheus scale
- 3:17What is Thanos
- 4:21Thanos splits Prometheus
- 6:10Thanos query and caching
- 9:22Thanos sidecar and Prometheus
- 11:05Migrating to Kubernetes
- 12:31Sharding Prometheus
- 14:43Thanos S3 costs
- 16:50Thanos compactor costs
- 19:22Handling range queries
- 21:50Local storage tier
- 22:53Storage with Rook and Ceph
- 23:3890% cost reduction
- 25:03Local storage costs and takeaways
related talks
30mIntro to Thanos: Scale Your Prometheus Monitoring With Ease
37mThanos: Prometheus in HA Mode with Persistent Storage
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry
15mScaling Observability Stack for AIOPS
21mThanos: Easier Than Ever to Scale Prometheus and Make It Highly Ava...
31mWhen Prometheus Can’t Take the Load Anymore