Who Watches the Watchers? From Closed Observability To...
summary
Prometheus, OpenTelemetry collectors, Cortex, Grafana and Tempo are treated as a distributed system: collectors add active traffic shaping (batching to 1024, tail sampling, memory limiter) so downstream calls drop from thousands/sec to tens/sec; recording rules, relabeling and a cardinality firewalling (per-target series limit 10k) defuse label storms; TSDB surgery (block duration 2h→15m, local retention → ~2h, WAL compression on) and remote-write tuning (queue 10k, shards 2k, retries 5→30) cut Prometheus memory from ~16GB to ~2GB and eliminate OOM crashes.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:483 a.m. observability nightmare
- 5:36Dashboards go blank during incidents
- 7:22Hidden Prometheus failure modes
- 13:24Telemetry as distributed system
- 15:10Fixes for observability reliability
- 25:00Mental models for monitoring
- 26:13Q&A
related talks
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
27mObservability with Prometheus and beyond
36mShould I use OTel (collectors), or is Prometheus good enough?
36mObservability with Prometheus and Beyond: Leveraging Cloud-native Technologies Outside of Cloud
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
49mObservability Day | Project Updates
25mDebugging the Pipeline: Observability for Your Observability with OTel + Prometheus
01h 19mIntro + Deep Dive: Prometheus