Anomaly Detection System for the CERN Cloud
summary
A production anomaly detection system for the CERN cloud ingests collectd metrics, runs a Spark ETL and Dockerized analytics pipeline orchestrated by Airflow, and publishes top-20 anomaly JSONs to Elasticsearch for Grafana dashboards. It combines Isolation Forest and LSTM-GRU autoencoder models via pyOD and TensorFlow, does inference in ~7–8 minutes on one VM, and was benchmarked on 40 hosts over 2 months (12,000 four‑hour windows, 2% anomalies). At 0.1% FPR the old threshold system has 8% TPR vs the new system 21% (at 4% FPR old 26% vs new 92%).
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:04Outline
- 2:24Large Hadron Collider & Detectors
- 3:12CERN Cloud Infrastructure at a Glance
- 4:38Current Anomaly Detection System
- 6:49Possible Approaches
- 8:42Anomaly Detection System - Library
- 10:05Anomaly Detection System - Architecture
- 10:48Technologies
- 12:30Anomaly Detection System - Deployment
- 15:41Customizable Data Analytics Core
- 16:18Anomaly Detection System - Results
- 18:04Traditional ML Methods
- 18:50Deep Leaming Methods
- 19:58Scoring Anomalies
- 21:14Figures of Merit
- 22:18Benchmark Dataset
- 23:44Summary
- 24:27Thank you for the attention!
related talks
35mAnomaly Detection for Cloud Native Storage
35mAnomaly Detection for Cloud Native Storage
29mAI-Powered Anomaly Detection in Cloud Data Pipelines
38mWatch Everything, Watch Anything: Anomaly Detection
29mREAL-TIME ANOMALY DETECTION MADE EASY
31mAnomaly Detection on OpenStack Logs using M.L.
29mAnomaly Detection
39mMastering real-time anomaly detection with open source tools