Resolving Outages Faster With Better Debugging Strategies
summary
A concrete workflow that replaces “look at dashboards” with dynamic query evaluation, arbitrary data joins, and exemplars/dynamic sampling finds a single bad replica in a fleet of thousands by drilling region → zone → instance, joins latency to kernel version to pin the culprit, and links to Zipkin traces for a real example of slow batches. Hypothesis testing drops from ~15 minutes to ~20 seconds in demos, works on Kubernetes-scale services, and warns about storage and dashboard sprawl costs.
AI-written summary · about
chapters (9)
- 0:00Introduction and scope
- 1:45Understanding system outages
- 3:37Debugging workflow and goals
- 5:33Tooling and setup
- 7:50Technique 1: Layered healing
- 12:12Technique 2: Data joints
- 18:30Technique 3: Using exemplars
- 22:56Debugging with traces
- 25:55Conclusion and best practices
related talks
39mResolving Outages Faster with Better Debugging Strategies
18mK8s Troubleshooting Demystified: Five Best Practices
34mChaos Debugging: Finding and Fixing Microservice Weak Points
35mObservability for Dask in Production
14mLeveraging Distributed Tracing in Kubernetes & Containerized Environments
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
21mDebugging at Scale—Going from Single Box to Production
26mDebugging Clojure Services Under Stress