Adding Observability to Distributed Systems
summary
A production email security service starts life with no observability, then adds generated transaction IDs on every message to reconstruct flows across ~80 microservices and Kafka; that homegrown trace-assembly over logs works but is hard to migrate. Uses OpenTracing-compatible tools—Zipkin, Jaeger, Elastic APM—and recommends sampling (Google’s Dapper example ~1/10,000) and exempting automation runs; a wall dashboard and per-customer SLIs (100k unique metrics, automation every 5 minutes) cut mean-time-to-detect dramatically. overloaded ELK for tracing
AI-written summary · about
chapters (22)
- 0:00Introduction
- 4:50Observability Venn Diagram
- 11:18Service Level Agreements
- 14:34Service Level Dashboard
- 16:55Reddit
- 18:10Metrics
- 23:16Example
- 25:56Logs
- 32:46Visualization
- 33:52Distributed Tracing
- 40:19Transaction Tracing
- 42:02Business Questions
- 45:09Collecting Metrics
- 47:57Open Standards
- 50:34Visualizing Tracing
- 51:46Tools
- 53:35Elastic APM
- 54:10Jaeger
- 55:10Zipkin
- 55:39Dependencies
- 56:06AWS Xray
- 56:54Code Example
related talks
01h 07m"Adding Observability to Distributed Systems"
49mImproving Observability of Distributed Systems
33mEffective observability in microservice architectures
42mDistributed Monitoring: How to understand the Chaos
48mAPI Observability by Design: Building Systems That Tell Their Own Stories
16mThe State of Observability
57mYour Distributed System is Talking to You: OpenTelemetry
42mFirst Steps with Observability: OpenTelemetry and You