Know your data: The stats behind your alerts
summary
Choice of central measure changes alerting: an arithmetic mean reads 13.33 req/s while a geometric mean compresses that to ~6.52 and a harmonic mean sits ~19.19 until a node spikes and the harmonic jumps to ~32, so use median for stable historical baselines and use mean for current-state comparisons. Sampling matters: combine head and tail (purposeful) sampling, keep ~15 minutes of full traces for forensics, and use OpenTelemetry + R calls (Datadog/Splunk/Elastic can call R, Honeycomb supports event retention) to model Weibull and exponential distributions for failure and arrival rates.
AI-written summary · about
chapters (5)
- 0:00Introduction and stats basics
- 12:45Understanding central tendency measures
- 22:52Sampling and data distribution pitfalls
- 30:41Common statistical mistakes in alerts
- 32:46Q&A on math and sampling
related talks
38mKnow your data: The stats behind your alerts
10mTurning data into leverage
38mMonitoring 101
40mNext-generation Alerting and Fault Detection
52mReal-time-first metrics
18mLook at Your Data
44mActionable insight with derived second order monitoring data
38mWatch Everything, Watch Anything: Anomaly Detection