OTel-y Oops: Learning From Our Observability Blunders
Talkat Observability Day 2025 · on Tuesday, 2025-04-01· at 15:55· Level 1 | Hall Entrance S10 | Room B·
summary
A Greenfield object storage project at Akamai over-instruments and floods the backend, producing a sudden spike (one incident shows ~38 million simultaneous series) and ultimately 300 million time series worth of noise that drives billing and outages. The pipeline uses OTel agents with a Kubernetes attribute processor that mislabels metrics behind an Istio sidecar, causing explosions of cardinality, VictoriaMetrics, Loki and Tempo are the consumers, and autoscaling plus removing agents from the mesh fixes the immediate failures.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:04Greenfield project with OpenTelemetry
- 2:48Early OTel setup issues
- 4:5638M time series explosion
- 10:14Logging stack overwhelmed
- 12:00Testing resilience early
- 14:08Misusing OTel receivers
- 16:14Crash telemetry gaps
- 19:47Lessons learned and benefits
- 20:50Summary
- 21:36Q&A
related talks
11mLightning Talk: OTel Me How to Build a Data Pipeline for Observability
52mGraphQL Observability with Elastic and OpenTelemetry
25mUnpacking Your OTel Baggage
30mPanel: Telemetry That Matters
22mOTel Sucks (But Also Rocks!)
45mFrom Noise To Signal: Building Smart Observability Pipelines With OpenTelemetry
48mAPI Observability by Design: Building Systems That Tell Their Own Stories
01h 00mObservability made painless: Go, Otel & LGTM stack