Online Change Point Detection Using Spark Streaming
summary
Scan Statistics are applied to streaming data using Spark Streaming and a homemade Python package that implements Bayesian online change point detection with a constant hazard and a t‑distribution prior. An accumulator Q keeps a bounded run‑length window across microbatches; a demo feeds NOAA daily temps one per second, produces posterior contour plots that flag abrupt shifts (threshold ~0.5), correlates those shifts with short‑lived sales spikes, and notes burn‑in and insensitivity to slow trends.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:50Why online change point detection matters
- 3:01Nonparametric and Bayesian algorithms
- 8:23Handling infinite data with windows
- 11:40Case study: weather and consumer behavior
- 13:08Demo: Spark Streaming and accumulators
- 15:34Detecting abrupt changes in weather
- 19:21Visualizing change points across US
- 22:11Q&A
- 25:09Comparing models and thresholds
- 27:06Closing remarks
related talks
01h 29msktime: time series anomaly detection, changepoint detection, segmentation
37mRealtime Bayesian A-B testing with Spark Streaming
28mUsing Changepoint and Bayesian Analysis to Drive Safety Improvements in Mining
47mDifferentially Private Change-Point Detection
43mDetecting Anomalies in Streaming Data
17mStopping Performance Regression via Changepoint Detection
32mPopmon: Population Shift Monitoring Made Easy
52mDevil in the Haystack