How Cheap Learning (not deep learning) Finds Faults in Distributed Systems
summary
Argues that small, fast statistical tools spot root causes in messy distributed stacks by measuring and comparing tagged distributions rather than training big models. Uses log‑scaled histograms for latency, t‑digest for accurate percentiles, simple count comparisons and tagging of dimensions (disk, node, table, job, version). Detects 1% or even 0.1% anomalous transactions within a minute at high rates and surfaces a few suspect disks or services. Implementations run at MapR and with large users like Netflix Google Microsoft and are practical and effective.
AI-written summary · about
chapters (15)
- 0:00Start
- 3:46What the Developer Thinks
- 5:01How the Data is Governed
- 6:56The Gap We Have to Cross
- 7:11Some complications
- 13:41An Example - 10 Million Disk Write Operations
- 15:50Good Latencies (synthetic data)
- 16:27Better View of Latency Spectrum for Disk Writes
- 17:57Modeling Inter-arrival Times
- 21:30The Power of Distributions
- 22:12Detecting Changes in Distribution
- 23:24System Outline
- 26:26The Power of Tagging
- 27:30Tagging Change
- 31:17Summary
related talks
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
52mDevil in the Haystack
09mDebugging Hadoop
49mShifting Safety Techniques To a Statistical World
12mModel Monitoring: Detecting and Analyzing Data Issues
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
29mHorrors of Distributed Systems
41mFinding the Needle in the Haystack or Troubleshooting Distributed Systems