Bootstrapping a Multi-DC Cloud Native Observability Stack
summary
Bootstraps a full observability stack across two datacenters that survives an entire campus failure by wiring Prometheus, Loki, Tempo, Alertmanager and Grafana to per-DC endpoints. Uses HashiCorp Consul prepared queries for DNS failover and Prometheus service discovery, Grafana backed by MySQL master‑master, Loki clients sending to both endpoints, Alertmanager clustered, and Tempo replicated via Grafana Agent remote_write to avoid single-DC KV limits. Author reports a working Vagrant demo and 10s scrape cadence.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:59Observability cotty Stack
- 2:53Metrics
- 3:55Prometheus Configuration
- 4:25PromQL
- 5:42Loki Configuration
- 7:57Tempo Configuration
- 9:09Dashboards
- 11:57Exploratory Usage
- 12:35Alerts / Alertmanager
- 13:28Alerts are queries too
- 14:26Multi-DC Requirement?
- 15:51Observability Stack HA?
- 17:38Consul Services
- 18:55Consul Prepared Queries
- 20:21Grafana HA, nginx config
- 22:10Grafana HA, Data Sources
- 24:52Grafana Agent
- 25:14Tempo/Traces HA
- 26:29Alertmanager/Alerts HA
- 27:22Observability Stack HA QED
related talks
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
22mAdding Observability to HashiCorp Nomad Applications with Grafana
40mBuilding a Scalable Observability Platform for Edge Compute Nodes
34mThe Grafana Stack: Building a Highly Available, Scalable Observability Platform
31mCloud Native Observability
35mFrom New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes
28mUnlocking Multi-Cloud Observability: The Case Study of...
23mImplementation of Unified Observability at Scale From Scratch