How Deep Systems Broke Observability
summary
Argues that modern microservice stacks are fundamentally deep systems (>4 layers) not just scaled-up puddles, citing customers with 10–2,000 services and Google examples where requests go ~30 layers deep. Presents a concrete approach: collect thousands of traces, automatically compare baseline vs regression, correlate tags (found a large_batch tag deep in the call graph) and surface a hot path four layers down. Traces are central — traces provide hot-path context and can reduce hypotheses by ~90% so operators find and fix the real bottleneck.
AI-written summary · about
chapters (8)
- 0:00Intro
- 1:30Scaling
- 3:10Deep Systems
- 8:40Observability
- 9:52Monolith
- 10:27Stress
- 15:01Traces
- 19:18Inventory
related talks
09mObservability for Deep Systems
33mEffective observability in microservice architectures
16mThe State of Observability
49mImproving Observability of Distributed Systems
38mThe Murphy's Laws of Observability
21mNavigating the Complexities of Service to Service Invocations: Deep and Brief Dive Into Causality
32mA Picture is Worth 1,000 Traces
31mUnderstanding Microservices with Distributed Tracing