Implementation of Unified Observability at Scale From Scratch
summary
Built a unified observability platform across 200+ sites handling 2,000 VMs, 10,000 network devices and 300 firewalls using Grafana Alloy in clustering mode with OpenTelemetry collectors, Loki and custom syslog/SNMP parsers; consolidation cuts infrastructure spend by six figures. Per-site collectors behind load balancers forward UDP syslog and SNMP, configs are managed via fleet tooling (Grafana Cloud/Ansible), recovery from a file‑descriptor leak and an AWS regional outage restores core function in hours, and ongoing work adds point‑in‑time config restore.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:06Why observability matters
- 4:15Infrastructure overview
- 5:39Observability components
- 7:05Per-site architecture
- 9:56Collector choice
- 10:17Network challenges
- 12:04Metrics labeling pitfalls
- 13:10Infrastructure and application observability
- 13:53Security observability
- 14:13Implementation issues
- 15:37Formats vs standardization
- 16:20Cost vs team resistance
- 17:25AWS outage lessons
- 19:12Backup strategies
- 19:53Observability as tier-one
- 20:36Results and benefits
- 21:41Q&A
related talks
44mGrafana Labs Customers: What We've Learned Building Observability at Massive Scale
45mDecentralizing SaaS Monitoring Using Open-Source Tools and Standards
28mMonitoring Stack was yesterday – Observability Platform at scale!
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
17mFrom noise to signal: piping your way to clear observability
50mIntuit Journey To Unified Observability at Scale: Challenges, Be...
36mObservability at Scale With Monitoring as Code...
22mUnified Observability