Declarative Multi-Cluster Monitoring with Prometheus
summary
Company runs per-namespace Prometheus instances and a federated data-center Prometheus, using the Prometheus Operator to declaratively deploy Prometheus, Alertmanager and ServiceMonitor CRDs so teams only supply selectors not configs. Operator automates upgrades across Prometheus 1.4→2.2.1 and encodes alerting/dashboard provisioning; federation gives a global view but has sticky-session issues, so they add Thanos sidecar to ship blocks to object storage and extend retention from one day to four weeks, keeping local alerting unchanged.
AI-written summary · about
chapters (17)
- 0:00Start
- 0:48Quick Prometheus intro
- 2:19Alert Evaluation
- 2:53What's a target?
- 3:46Kubernetes Discovery
- 4:58Discovery via kubernetes Services
- 6:30Target overview
- 6:48A Kubernetes config (1/n)
- 8:37Kubernetes native configuration
- 11:10Declarative everything
- 12:13Declarative Grafana Dashboards
- 15:00Prometheus per Namespace
- 15:46Prometheus per Cluster
- 16:08Service Monitor
- 17:07Drawbacks with current Federation
- 27:14Thanos support in Prometheus Operator
- 27:25Summary
related talks
31mDeep Dive into the Prometheus Operator Ecosystem
19mKeynote: Monitoring Kubernetes Clusters with Prometheus
32m“Gluing it all together” aka building a platform around Prometheus
25mMulti-Cluster Observability: What you need to survive with a fleet
28mPrometheus Operators in the Wild
24mPractical Kubernetes Monitoring with Prometheus
37mMonitoring Kubernetes with Prometheus
46mLeveraging Prometheus Service Monitor and Pod Monitor for Enhanced Kubernetes Monitoring