Telemetry is Not Having to Hope
summary
A personal story shows telemetry catching a real bug: clock drift causing partial login failure that logs and EC2 metrics missed, so the workflow moves from reactive firefighting to proactive evaluation. The stack centers on Prometheus exporters plus custom instrumentation to record latencies, forks in the road decisions and backpressure, and to compare changes like switching compression algorithms where A is faster but B costs more disk. The outcome is an iterative, monitoring-driven workflow: instrument first ask questions later.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:30My Story
- 3:13Clock Drift
- 4:24Telemetry is abstract
- 4:49Latencies
- 5:12Forks in the Road
- 5:54Back Pressure
- 6:14Prom Con
- 6:31Everything is Consumable
- 7:28Cost of Complexity
- 8:06Open Source
- 8:31Cost
- 11:08Low Barriers
- 12:21Prometheus
- 13:10Monitoring
- 14:48Proactive
- 18:03iterative
related talks
39mTelemetry ...and metrics for all
12mHarnessing Chaos Engineering to Build Robust Systems
05mTelemetry Pipelines: Your Recipe for Success
27mTelemetry: By The Numbers
30mPanel: Telemetry That Matters
27mThe Signal in the Storm: Practical Strategies for Managing Telemetry Overload
30mTelemetry: Industry Status, Challenges, and OSS Opportunities
20mTelemetry Pipelines: Never Gonna Let You Down