Anomaly Detection for Platform Logs using Embedded Context
summary
Claims an end-to-end pipeline that ingests millions of logs per day, enriches entries with deployment user and region metadata, analyzes streams with autoencoder + transformer embeddings and classical models like isolation forest, and acts through dashboards, Slack and automated mitigations. Implementation uses OpenTelemetry/Fluentd collectors and a Kafka message bus for durable streaming, plus specialized collaborating agents (auth perf network). LLMs are useful for summaries but fail at real‑time scale and raise cost and privacy concerns.
AI-written summary · about
chapters (10)
- 0:00Introduction to Log Analysis
- 0:35Challenges in Log Management
- 3:09Traditional Approaches and Their Limitations
- 4:57Introducing Embedded Context Agents
- 7:13Architecture of Embedded Context Agents
- 12:53Detection Core and ML Models
- 13:26Multi-Agent Collaboration
- 17:56Benefits and Challenges of Embedded Context Agents
- 18:53Future Directions and Use Cases
- 20:18Conclusion
related talks
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
10mML-based Log Analysis for Faster Debugging
31mAnomaly Detection on OpenStack Logs using M.L.
26mML Monitoring & Observability
35mAnomaly Detection for Cloud Native Storage
35mAnomaly Detection for Cloud Native Storage
01h 21mBuilding a ML Model Observability pipeline