Spike Detection in Alert Correlation
summary
LinkedIn builds a call graph of microservice dependencies and applies a modified zscore (median absolute deviation based) to each service metric time series to mark short lived spikes, then combines results across graphs and thresholds (five consecutive spike points and ≥70% of graphs) to decide real incidents. The pipeline pulls alerts from AutoAlerts, filters spikes, posts Slack recommendations and an API, and reduces oncall toil by about 30–40% while classifying ~36% spikes versus ~63% real alerts.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:29Background: Quick 1 Introduction of Linkedin Stack
- 1:32Linkedin Stack Under the hood
- 3:23Finding Needle in a haystack
- 5:07Alert Correlation A framework that automates the alert correlation process to identity unhealthy microservices
- 6:35Alert Correlation Slack Recommendations
- 7:57A Real Issue
- 8:50A Spike
- 9:49Correlation does not mean Causation
- 10:18Problem Statement: Finding the "right" needle in a needlestack
- 11:49Modified Z-Score For Outlier Detection
- 12:39MAD (Median Absolute Deviation)
- 14:05A Simple Example
- 15:09Spike Detection Challenges
- 18:08Results: Spike vs Real
related talks
30mReducing MTTR and False Escalations: Event...
46mEvent Correlation: A Fresh Approach towards Reducing MTTR
37mWeathering the Storm: How Early Warnings Save the Farm
40mNext-generation Alerting and Fault Detection
20mNew Approaches to Reduce Alert Noise AIOps
11mAssaf Resnick (BigPanda)
26mAIOps: New Approaches to Reduce Alert Noise
19mAutomatic Metric Screening for Service Diagnosis