Debugging at Scale—Going from Single Box to Production
summary
Covers concrete techniques used at Bing and Cortana: Zipkin-style distributed tracing with trace ID based log retrieval, Kafka + Storm pipelines to compute metrics in near real time metrics (two to three minutes), and an on‑demand profiler called Beeper plus a StampManager log collection agent. Shows log‑playback, traffic forking, and clustering/classification of logs to pinpoint failures; cites tens of thousands of weekly A/B experiments and examples where a single machine outlier or SMART disk errors caused production degradation. Demo Hololens prototype works as a visual on‑call aid.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:51Context
- 5:45Concepts
- 8:53Distributed tracing
- 10:13Design for reproducibility
- 11:01Machine learning
- 13:49Near realtime metrics
- 15:48Closing thoughts
related talks
31mFrom Alerts to Insights: Performing Trace-Based Causation at Scale
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
25mFrom Blind Spots to Total Vision: O11y at Massive Scale
30mDebugging at Scale Using Elastic and Machine Learning
52mModern Observability and Event Driven Architectures
01h 00mSystems @Scale Summer Q&A Week 4
24mDistributed Tracing : New DevOps Foundation.
45mTracing: A journey to tactical insights