Debugging the Pipeline: Observability for Your Observability with OTel + Prometheus
summary
Shows how to monitor OpenTelemetry Collector + Prometheus with concrete signals: exporter queue length, Q sizes and send_fail counters, process CPU/memory, receiver accepted/refused counts, and scrap_samples to detect silent drops. Explains Target Allocator failure modes like target allocator stops assigning and uneven assignment, remote_write lag and dropped samples, and configuration pitfalls that caused a reproduced 30% data loss example. Recommends a collector-only health dashboard and specific metrics to alert on before dashboards go blank.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 1:51The silent failure problem
- 4:31Pipeline anatomy and failure modes
- 6:44Debugging the collection stage
- 9:30Collector metrics and health
- 12:06Troubleshooting and best practices
- 17:19Target allocator and stability
- 19:17Remote write and final takeaways
- 22:15Q&A and concluding remarks
related talks
20mTelemetry Pipelines: Never Gonna Let You Down
11mLightning Talk: OTel Me How to Build a Data Pipeline for Observability
26mDecoding OpenTelemetry(OTel) Collector: A Deep Dive into Architecture
24mBuilding Resilient Telemetry Pipelines: Mastering the OpenTelemetry Collector's Per...
37mShould I Use OTel (collectors), or Is Prometheus Good Enough?
13mConnecting Prometheus and OpenTelemetry Data for Faster Troublesho...
50mGuide to Observability with OpenTelemetry
24mLeveraging OpenTelemetry for Your Prometheus Pipeline