Finding Needles in a Growing Haystack
summary
An architecture that turns terabytes of trading logs and ~100k candidate time series into daily alerts using online Bayesian updating and simple conjugate models (beta for rates, Dirichlet/Gaussian-process ideas for quantiles and functions). Data lands in REST APIs and MongoDB as typed JSON, reports run on a Mesos + Aurora cluster with ~800 cores, Python jobs, and multiprocessing; results surface as JSON and dashboards with 95% credible intervals and detected outliers so the system runs in production.
AI-written summary · about
chapters (8)
- 0:00Finding needles intro
- 3:09What market making is
- 8:26Bayesian stats solve analysis
- 13:48Bayesian nonparametric scaling
- 18:22Scaling to production
- 27:33Monitoring in production
- 30:23Stack technologies
- 30:44Closing remarks
related talks
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
52mDevil in the Haystack
48mHyper Speed! When Big Data Blooms
24mHunting Criminals with Hybrid Analytics
19mBig Data
17mOptimizing Data-Driven Decisions
51mReal-time Fraud Detection: Challenges and Solutions
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning