Mastering Distributed Tracing: Insights from Building a Platform from Scratch
summary
Built a distributed-tracing platform around Hypertrace, Jaeger/OpenTelemetry, Kafka and Pinot, ingesting 300K spans/sec average (peaks ~800K) and about 18 TB/day, with 100% sampling and seven-day retention after cutting 30 days. The platform adds an Explorer query UI, alerting that reuses existing pager workflows, dashboards, deployment markers, and a quality-index that finds noisy or sensitive tags; storage moves to StarTree + S3 and a one-day SSD + six-day S3 tier lowers costs and TCO ~10–15%, and MTTD/MTTR measurably improves.
AI-written summary · about
chapters (8)
- 0:00Introduction to distributed tracing
- 7:42Goals and benefits of tracing
- 14:02Making tracing user-friendly
- 21:50Ensuring trace quality and security
- 28:12Instrumentation and adoption challenges
- 32:50Cost optimization and retention strategies
- 40:14In-house vs. SAS decision
- 43:50Quantifying tracing platform value
related talks
24mDistributed Tracing : New DevOps Foundation.
29mIntroduction to Distributed Tracing
26mDistributed Tracing: All the Warning Signs Were Out There!!!
33mDistributed Tracing - All The Warning Signs Were Out There!
34mReduce Complexity with Distributing Tracing
34mDistributed Tracing, Lessons Learned
31mFrom Alerts to Insights: Performing Trace-Based Causation at Scale
42mFounder/CTO Perspectives: The Future of Distributed Tracing