Pintrace: A Distributed Tracing Pipeline
summary
Pinterest builds a Zipkin-based pipeline called PinTrace that collects spans via local logs → Kafka (using singer) → a Spark job into Elasticsearch, handling ~50,000 hosts and sampling ~1% of traffic. Instrumentation spans Python, Java (Finagle) and mobile; a manual tracing effort earlier cut search latency ~50%, while fixes from traces shave 20 ms (duplicate calls), 5–10 ms (parallelize memcache), and ~100 ms (reorder an ad call). The pipeline works and teams adopt it for debugging, dependency graphs and end-to-end CDN-to-client latency.
AI-written summary · about
chapters (25)
- 0:00Distributed tracing at Pinterest
- 1:14Why tracing became essential
- 4:27Manual to automated tracing
- 5:55Building Pinpoint and early attempts
- 9:28What tracing is and how it works
- 10:33Five challenges in tracing systems
- 11:14Challenge: Instrumentation across languages
- 15:05Challenge: Report and aggregation pipeline
- 16:52Challenge: Deploying at scale
- 17:35Challenge: Tracing costs with sampling
- 18:59Challenge: Low-latency services and blame
- 20:06Trace processing and storage
- 21:12Customizing Zipkin UI and analysis
- 22:58Developer education and adoption
- 24:02Understanding request execution
- 26:59Debugging with traces and duplicates
- 28:29Debugging slow memcache calls
- 30:17Tuning systems with trace analysis
- 31:19Metrics and traces for debugging
- 33:26Service dependency graphs
- 34:52End-to-end tracing mobile to DB
- 36:40Tracing through CDN for latency
- 38:30Future tracing applications
- 41:00Lessons from building PinTrace
- 43:07Q&A
related talks
50mDistributed Tracing in the Wild
36mDistributed Tracing: From Theory to Practice
42mAn introduction to distributed tracing and Zipkin
29mDistributed Request Tracing
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
45mTracing: A journey to tactical insights
34mSlackTrace: A New Tracing Tool
34mDebugging Your Services with Distributed Tracing