Log messages processing using NLP tools
summary
A complete monitoring pipeline uses Drain clustering to turn freeform logs into templates and cluster IDs, then fits a multivariate Hawkes process with an exponential kernel to learn a 203×203 adjacency matrix of triggers; the dataset is ~33 million log lines, one cluster has ~700k representatives, fitting takes about 10 minutes to fit, most baseline mus are zero and a few alphas dominate, thresholding edges yields a 41-node directed graph that shows second-scale lead/lag correlations and actionable causal chains.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:55Key use cases for log analysis
- 5:06Clustering logs with Drain Free
- 7:13Hawkes process for log intensity
- 10:53Processing 33 million Spark log lines
- 13:49Correlating clusters with Hawkes
- 18:54Visualizing cluster impacts
- 23:59Challenges and future work
- 25:24Conclusion
related talks
31mLearning Latent Events from Network Message Logs
30mFast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
30mDebugging at Scale Using Elastic and Machine Learning
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
43mLogAn: Large-scale Log Analysis with Small Language Models
48mHunting Stealth Adversaries with Graphs & AI
24mHow to numerically represent semi-structured log data for...