Tracing Service Dependencies at Salesforce
summary
Salesforce processes ~300 million spans per minute and ~100 million unique traces per minute through a Flink-based streaming pipeline that builds in-memory trace contexts, emits aggregated dependency edges into Kafka and stores them in a Druid backend for queries and visualization. The system samples ~10% of traces, retains aggregated edges for 30 days, detects surprising cyclic dependencies, surfaces per-edge health and latency, and costs roughly an extra 5% over prior tracing infrastructure while driving much higher tracing adoption.
AI-written summary · about
chapters (12)
- 0:00Introduction to tracing
- 1:02Microservices dependency challenges
- 3:51How distributed tracing helps
- 5:18Tracing platform at Salesforce
- 7:05Distributed tracing semantics
- 9:52Building dependency graphs
- 13:03Visualizing service dependencies
- 16:13Transaction dependency mapping
- 20:29Increasing trace coverage
- 21:51Detecting broken traces
- 23:38Rollout impact and feedback
- 24:21Q&A session
related talks
29mIntelligent Log Analysis and Real-time Anomaly Detection @ Salesforce
34mDistributed Tracing, Lessons Learned
45mTracing: A journey to tactical insights
50mDistributed Tracing in the Wild
41mTracing Real-Time Distributed Systems
54mLatency and Event Tracing with Spring Cloud Data Flow
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
33mDistributed Tracing - Implementing Latency Analysis for microservices on Cloud Foundry