From Alerts to Insights: Performing Trace-Based Causation at Scale
summary
Engineers build a streaming pipeline that reconstructs traces in memory from span messages, Go and Kafka power the consumer, and per-trace depth‑first searches extract error/latency chains which are counted and surfaced with up to 5 exemplars per path. It runs on a 20% head sample (about 8.3M spans/sec), partitions by trace ID, waits ~2 minutes for traces to complete, writes aggregated paths to Azure Data Explorer, and reduces metric-shaped red herrings by returning anomalous causal paths for triage.
AI-written summary · about
chapters (12)
- 0:00Introduction and team mission
- 2:57Prior approaches to incident analysis
- 6:33Limitations of metric-based correlation
- 9:03Real-time trace aggregation pipeline
- 13:25Building the pipeline with Go
- 16:42Pipeline optimization challenges
- 18:32Measuring pipeline precision
- 21:02Expanding to client-side traces
- 22:46Supporting deeper levels of blame
- 24:31Improving anomaly detection
- 25:11Conclusion and recommendations
- 25:54Q&A session
related talks
45mTracing: A journey to tactical insights
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
34mDistributed Tracing, Lessons Learned
24mData analytics with distributed tracing data
21mDebugging at Scale—Going from Single Box to Production
51mNavigating Distributed Traces
49mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
32mA Picture is Worth 1,000 Traces