Debugging at Scale Using Elastic and Machine Learning
summary
A production pipeline ingests Kafka/Storm traces and user impressions, runs distributed tracing via Zipkin, tokenizes logs into sentences and key/value pairs, and applies ML classifiers and clustering in a system called flycatcher to surface service outliers across tens of thousands of machines. The stack writes to Elasticsearch/Kibana, uses Twitter’s AnomalyDetection for time series, flags detects negative phrases within 5 minutes and exposes implicated components in ~10–15 minutes; GDPR and log-retention for privacy require careful handling.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:59Admiral Grace Hopper
- 2:36Debugging Philosophy
- 3:32Debugging is Hard
- 4:06Who am I
- 5:26Broken Tools
- 6:11Traditional Tools
- 7:10Solutions
- 7:28Error Messages
- 9:21Distributed Tracing
- 10:08Zipkin
- 11:17Machine Learning
- 12:38Linear Algebra
- 13:04Log Relevant Organization
- 14:10Classifiers
- 15:27Clustering
- 19:23Bringing it together
- 21:49Privacy
- 25:00Closing Thoughts
- 26:19Questions
related talks
45mTracing: A journey to tactical insights
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
24mData analytics with distributed tracing data
01h 02mAdding Observability to Distributed Systems
21mDebugging at Scale—Going from Single Box to Production
10mML-based Log Analysis for Faster Debugging
29mTune Your Microservices by Learning from Traces