Lightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
summary
Distributed tracing costs explode when every span is sent to an external APM and storage/egress scale with requests; pragmatic solution uses custom sampling to drop middleware/formatting spans, an OpenTelemetry tail-sampling processor in an OpenTelemetry Collector to accept traces only on latency or 5xx signals, and local attribute filtering to avoid provider-UI-dependent fields. Concrete results: a simple sampler drops middleware and format spans (10→6 example) and reduces a 87-span trace to 53 — roughly ~40% span reduction and much lower bills.
AI-written summary · about
chapters (10)
- 0:00Distributed tracing costs
- 1:03Debugging in microservices
- 2:25Tracing bills grow with scale
- 3:30Mixed sampling experiment failure
- 4:56Filtering duplicate span attributes
- 6:00Tail sampling limitations
- 7:23OpenTelemetry custom sampling
- 8:50Custom samplers reduce spans
- 10:36OpenTelemetry tail sampling
- 11:59Summary
related talks
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
33mReal-World Sampling – Lessons Learned...
21mLightning Talk: Distributed Tracing Without the Pain!
09mGraduated Project Lightning Talk: Using Trace Data for Monitoring & Alerting of Appl...
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
27mTricked Out Traces
34mReduce Complexity with Distributing Tracing