Anomaly Detection in Telecom with Spark
summary
Architecture and implementation details for realtime telecom anomaly detection using Spark Streaming with a Kafka-like broker (MapR Streams / Apache Kafka), a NoSQL store and a web UI consuming alert topics; demo simulates 6 antennas and 100 devices, and notes operators see up to 3 terabytes per day or millions of messages per second. Algorithms are simple thresholds, percentiles or clustering applied on aggregated call-rate windows; anomalies are detected as sustained drops or spikes and emitted back as Kafka alerts. Tradeoff: micro-batching (seconds latency) vs lower-latency engines like Flink.
AI-written summary · about
chapters (7)
- 0:00Introduction and speaker background
- 1:51What is anomaly detection
- 10:18Building the anomaly detection model
- 20:59Streaming data architecture
- 27:48Live demo setup and overview
- 37:06Key technologies for real-time processing
- 41:09Q&A session
related talks
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
19mBig Data
39mMastering real-time anomaly detection with open source tools
52mDevil in the Haystack
27mDataEngBytes 2023 - MEL-T1-08 - Troy Sellers
30mMassive Scale Anomaly Detection Framework
48mPaths to anomaly detection w/ TIBCO data science, streaming on AWS