Distributed Tracing: All the Warning Signs Were Out There!!!
Talkat Observability Day North America 2024 · on Tuesday, 2024-11-12· at 15:55· Salt Palace | Level 2 | 255 E·
summary
eBay reports 6.5M spans/sec with 14-day retention and solves adoption by shipping OpenTelemetry instrumentation through a mandatory managed framework, plus service-mesh and mobile hooks; storage uses multi-cluster ClickHouse multi-cluster and per-cluster collectors with a gateway. Cost control relies on 2% head sampling plus exemplar harvesting and an exemplar-driven tail sampling flow that copies interesting TraceIDs into persistent tables; main failures are broken context propagation, collector memory limits for tail sampling, and low-traffic APIs needing 100% sampling via Jaeger policies.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:40What is Distributed Tracing
- 2:24The Four Point Problem
- 4:35Open Telemetry
- 5:04Free Instrumentation
- 5:32Adopt Open Telemetry
- 6:31Optimize for Mass Adoption
- 7:53Make Instrumentation Optional for Developers
- 8:48Build the Platform
- 9:57A Typical Cluster
- 11:17Optimizing for Cost
- 12:26Sending Less
- 13:12No More Redundancy
- 14:10Tail Sampling
- 14:20Usage
- 14:43Waterfall
- 14:57Context Propagation
- 17:33Performance Tuning
- 18:04A New Path
- 18:15A Simple Architecture
- 19:56True Tail Sampling
- 20:28VJ Sampling
- 22:11Critical Paths
- 22:55Aggregate Trace Analysis
- 24:06Was it all worth it
related talks
33mDistributed Tracing - All The Warning Signs Were Out There!
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
34mReduce Complexity with Distributing Tracing
57mYour Distributed System is Talking to You: OpenTelemetry
33mReal-World Sampling – Lessons Learned...
32mDistributed Tracing in Action: Our Journey with OpenTelemetry
41mTracing Real-Time Distributed Systems
01h 02mOpenTelemetry Tracing — Real Challenges in Complex Systems