Streamlining Telemetry Data
summary
An observability pipeline is built that routes high‑cardinality telemetry from hotel collector through the Mesmo pipeline into Prometheus and Grafana alternatives, with a data profiler that reports unique values, field sizes and drift over time. Daily ingress falls from about a terabyte a day to roughly 300 GB, a 60% reduction. Tag‑cardinality processors can be deployed in the pipeline to drop or limit exploding metrics and prevent recurring cost and noise spikes.
AI-written summary · about
chapters (6)
- 0:00Introduction to the Cardinality Problem
- 0:38Initial Setup and Challenges
- 2:05Implementing the Solution
- 3:02Data Profiling and Monitoring
- 5:00Future Plans and Recommendations
- 6:55Q&A Session
related talks
05mTelemetry Pipelines: Your Recipe for Success
17mFrom noise to signal: piping your way to clear observability
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
06mObservability Diet: Your 5-Step Plan To Trim the Data Fat
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
30mPanel: Telemetry That Matters
20mAggregating Metrics In-Flight: Challenges, Opportunities