Demystifying Anomaly Detection in OpenSearch
summary
A working pipeline sends systemd journal logs into OpenSearch via fluent-bit systemd plugin, indexes them (running in Docker on localhost), and builds an anomaly detector that uses a counted metric (message count feature) over a configurable window and delay. The detector runs a Random Cut Forest model (Random Cut Forest), flags high anomaly grade with rising confidence, and triggers a per‑query monitor alert channel. It quickly surfaces an SSH brute‑force burst in minutes, so the setup demonstrably works.
AI-written summary · about
chapters (10)
- 0:00Demystifying Anomaly Detection
- 2:37The Project and Community
- 3:30Let's build one of these together!
- 5:28Ingestion
- 10:16Creating a Detector
- 15:21Configuring your Model
- 21:08The Panel Revisited
- 26:21Alerts require a Channel
- 27:20Alerting Options
- 30:36Come play with me!
related talks
42mObserve Smarter, Not Harder: Scaling AI-Powered Observability With O...
22mFrom Telemetry to Insight: Building AI-Powered Observability Pipelines with OpenSearch
18mFrom Logs To Signals: Rethinking System Monitoring With OpenSearch
19mThe Rise of AI in Anomaly Detection
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
26mAnomaly Detection System for the CERN Cloud
38mWatch Everything, Watch Anything: Anomaly Detection
46mOperational Logs and Event Management for HPC S...