Learning Latent Events from Network Message Logs
summary
Takes 97 million messages with about 1,000 message types and decomposes logs into episodes using a hierarchical change-point detection step that runs in linear time, then maps episodes to documents and applies LDA/topic modeling to recover ~15–25 underlying event distributions. Proves consistency and sample complexity, compares results to clustering and small-scale Bayesian inference, and shows the pipeline scales to the full dataset, returning event start/end times and per-event message distributions.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:56Example
- 5:21Modelling
- 12:09Topic Modeling
- 13:28LD Algorithm
- 15:20What are Documents
- 15:54What are Episodes
- 17:23What is the Algorithm
- 18:25The Basic Idea
- 19:14The HighLevel Idea
- 20:15Applying LD
- 20:34Analysis
- 20:48Consistency
- 22:05Sample Complexity
- 22:53Clustering Type
- 24:37Bayesian Inference
related talks
27mLog messages processing using NLP tools
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
21mAnomaly Detection for Platform Logs using Embedded Context
10mML-based Log Analysis for Faster Debugging
24mHow to numerically represent semi-structured log data for...
30mFast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
26mNLP for security log analysis : Learning to crawl before you run