Real-World Sampling – Lessons Learned...
summary
Pismo reduces observability spend by about 80% cost reduction by routing 100% of traces through an OpenTelemetry Collector using tail sampling policies that keep 100% errors/high-latency, exclude selected services, and probabilistically sample others. Implementation uses a 5s decision window, config to hold ~120,000 traces in memory, handles peaks near 250k spans/sec, and produced ~39.6k sampled spans (~16% of spans). Trade-offs: large stateful collectors (25 CPU / 56 GB), horizontal scaling, possible lost late spans, and extra operational complexity.
AI-written summary · about
chapters (10)
- 0:00Real-world sampling intro
- 1:24What sampling is and why it matters
- 3:11Head-based vs tail-based sampling
- 9:27Why Pismo chose tail sampling
- 11:38Pismo’s tail sampling implementation
- 16:43Collector scaling lessons
- 18:53Probabilistic sampling accuracy
- 20:41Detecting incidents with sampling
- 21:48Cost savings and next steps
- 22:57Q&A
related talks
27mTricked Out Traces
39mWhy, How to, and Issues: Tail-Based Sampling in the OpenTelemetry Collector
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?- 12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
35mRefining Systems Data without Losing Fidelity
35m99.99% of Your Traces Are (Probably) Trash
26mDistributed Tracing: All the Warning Signs Were Out There!!!