Observability at Scale With Monitoring as Code...
summary
An enterprise observability platform built with Grafana, Prometheus (AWS Managed Prometheus), and Grafana Tempo stores traces to S3 and enforces self-service via Monitoring as Code: a Typescript AWS CDK L3 construct plus per-repo pipelines deploy dashboards, alerts, and on‑call configs. It handles 100+ EKS clusters, 3,000+ containers per environment, ~16 billion metric points/day with 200k metrics/sec injection, and sustained trace loads up to 40k spans/sec with tests to 80k spans/sec. Pragmatic fixes include batch-to-microservice aggregation and Tempo gossip isolation.
AI-written summary · about
chapters (20)
- 0:00Observability at scale intro
- 1:26Company context
- 3:56Requirements and challenges
- 6:45NextGen platform overview
- 8:30NextGen standards and tools
- 11:21Platform architecture details
- 15:15Trace collection workflow
- 19:36Monitoring as code intro
- 19:57Building monitoring as code
- 22:08Self-service observability
- 22:31Demo: monitoring as code
- 24:17Dashboards and alerts
- 25:00Beyond app metrics
- 25:24Challenges and lessons
- 25:46Metrics scaling issues
- 26:51Multi-tenancy tracing problems
- 28:20Trace context loss in CDC
- 30:27Current scale and achievements
- 31:29Summary and empowerment
- 31:56Q&A
related talks
56mFull-stack observability and application monitoring with
25mGrafana as Code-An Open Source Observability Solution
27mMonitoring as Code: Managing your Observability Assets the Cloud-Native Way
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
27mFrom Zero Visibility to Full Observability with Grafana
31mCharting New Frontiers: The Evolution of Observability in Cloud Environments
28mMonitoring Stack was yesterday – Observability Platform at scale!
27mObservability with Prometheus and beyond