Monitoring multiple Kubernetes Clusters with Thanos
summary
Setup monitors 20+ Kubernetes clusters (some with ~1 TB JVM memory) using Prometheus plus Thanos sidecar for grpc queries and optional Thanos receiver for push ingestion. Metrics get pushed to an S3-backed store and a stateless Thanos store serves long-term data; a compactor deduplicates and downsamples blocks. Sharded receivers provide HA by replicating samples to N replicas; query-time dedupe hides duplicates. Relabeling enforces tenant labels but currently needs receiver restarts to pick up changes. The deployment runs a year without data loss.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 1:04Challenges with Prometheus
- 3:50Defining monitoring constraints
- 6:06Customer case study
- 10:33Introduction to Thanos
- 14:14Architecting with Thanos
- 16:58Implementing push-based metrics
- 21:23Redundancy and deduplication
- 24:46Multi-tenancy and conclusion
related talks
37mThanos: Prometheus in HA Mode with Persistent Storage
26mAgile Monitoring At Scale With Thanos
25mMulti-Cluster Observability: What you need to survive with a fleet
30mIntro to Thanos: Scale Your Prometheus Monitoring With Ease
26mPrometheus That Scales
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
24mTaming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...
24mPractical Kubernetes Monitoring with Prometheus