Tracing the Journey into Distributed Tracing
summary
Large-scale org rollout of distributed tracing targets concrete gaps: causality, discoverability and alert noise. Team runs a proof-of-concept, chooses a third-party backend for open tracing compatible instrumentation after load tests, and attaches the rollout to cyber week load tests to win sponsorship. Rollout is driven by an SRE team of five + program manager, instruments 3–4,000 services selectively around critical user operations, builds alerting, SLO, service diagrams and traffic estimators, and makes tracing the primary incident tool within a year.
AI-written summary · about
chapters (15)
- 0:00Distributed tracing journey intro
- 1:27Legacy monitoring problems
- 4:41Why tracing became solution
- 5:43Evaluating tracing tools
- 7:51Decision to buy tool
- 8:53Forming SRE team
- 10:37Cyber week adoption
- 13:05Rollout plan pillars
- 14:53Critical user operations
- 16:17Engineer workshops
- 17:42Adoption momentum
- 19:48Success stories promotion
- 21:12Overcoming resistance to new technology
- 22:55Post-rollout impact
- 23:59Key learnings from journey
related talks
29mHow we tricked engineers into utilizing distributed tracing
38mDistributed Tracing in Large Software Systems
59mAn introduction to distributed tracing
11mKeynote: Distributed Tracing - The Struggle is Real
31mUnderstanding Microservices with Distributed Tracing
26mDistributed Tracing FAQ
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
34mReduce Complexity with Distributing Tracing