Tracing is More Than Traces: The Insights in Trace Aggregates
summary
Aggregating thousands of LightStep traces and running statistical correlation on latency histograms surfaces the real causes of tail latency: resource contention on mutexes and specific high‑cardinality tag values. A donut‑service demo shows p99 and p99.9 analysis identifying a single server/customer and a fryer/topper mutex by a unique lock ID tag; chocolate orders hold the lock longer, so splitting locks reduces tail latency. This uses trace aggregates to compute the critical path and relies on open telemetry‑style portable traces.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:25Quick review
- 2:08A single trace
- 4:17Trace aggregate analysis
- 6:43Correlation based on statistical analysis
- 10:01Latency anomalies in context of service infrastructure
- 13:32Critical path analysis - resource contention
- 16:02Resource contention analysis
- 18:42Example: mutex contention-terminology
- 19:39Demo time
- 26:45What was this talk about?
related talks
32mA Picture is Worth 1,000 Traces
34mReduce Complexity with Distributing Tracing
33mOpenTracing and Containers: Depth, Breadth, and the Future of Tracing
54mUnderstanding Performance w/DTrace (While The Customer Yells)...
19mKeynote: OpenTracing and Containers: Depth, Breadth, and the Future of Tracing - Ben Sigelman
51mNavigating Distributed Traces
34mThe Holy Grail of Systems Analysis: from What to Where to Why
46mHow to Properly Blame Things for Causing Latency