LogAn: Large-scale Log Analysis with Small Language Models
summary
LogAn analyzes GB-scale log dumps by first applying log templatization (Drain algorithm) to collapse 38K lines into a dozen templates, then runs CPU-friendly Small Language Models to tag templates into six golden signals (availability, error, latency, saturation, traffic, info). It runs in production since March 2024 across 70 products, processes >2,000 tickets, claims 300+ man hours saved, cuts token load from 1.7M to a small representative set and drops inference latency from ~3,000s to ~12s.
AI-written summary · about
chapters (21)
- 0:00LogAn introduction
- 1:25RedHat adoption
- 2:06Log analysis challenges
- 4:56Log analysis evolution
- 8:04LLMs and log data
- 9:49Challenges and limitations of using LLMs for log analysis
- 11:56Problem and approach
- 13:41IBM research
- 15:50Log templatization
- 17:35Drain algorithm
- 24:18Templatization impact
- 25:43Semantic extraction
- 27:08Golden signals
- 28:30Three-stage pipeline
- 30:14Live demo on OpenSSH
- 33:42Analysis results
- 34:48Anomaly detection
- 35:31Performance metrics
- 36:53Open source features
- 37:38Roadmap
- 38:06Q&A
related talks
30mFast Log Analysis Made Easy by Automatically Parsing Heterogeneous Logs
10mML-based Log Analysis for Faster Debugging
37mChallenges and Pitfalls of Applying GenAI in Monitoring
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
27mLog messages processing using NLP tools
06mObservability for Large Language Models
32mMonitoring in the era of Generative AI, LLVMs, and embeddings
40mLarge-Scale Logging Made Easy