Stopping Performance Regression via Changepoint Detection
summary
Bloomberg applies Bayesian online change point detection to latency time series and finds a multi-day regression that spreads across machines by comparing before and after metadata on each host. A key signal is the work time metric which predicts runaway latency. Peak message volume grows to 490 billion messages per day, complicating noise and expected shifts like market open and maintenance. The project surfaces 12,000 alerts initially, so engineers build preprocessing in Graphite, tagging, true/false labeling and UI filtering to reduce toil and close the feedback loop.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:19Ticker plant role
- 2:41Why latency matters for Bloomberg's clients
- 4:24Detecting regressions
- 5:31Changepoint detection in action
- 7:39Real-world latency complexity
- 9:48Bayesian online changepoint detection
- 10:53Handling expected changes
- 13:26Triager usability fixes
- 15:36Implementation recommendations
- 16:41Closing
related talks
16mUsing Change Point Detection to Fight Noisy Benchmark Results
41mTracing Real-Time Distributed Systems
28mUsing Changepoint and Bayesian Analysis to Drive Safety Improvements in Mining
28mOnline Change Point Detection Using Spark Streaming
12mHistory-based Latency Prober Tuning
31mFinding Needles in a Growing Haystack
26mML Monitoring & Observability
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...