Data Analytics-Based Detection At Scale
summary
A next-generation analytics platform replaces a legacy CM with a Flink-based distributed rules engine reading from Kafka, stores raw events in Athena and runs TensorFlow models in AWS serverless services; it processes 60,000 events per second, captures ~6.2 TB/day, and uses an event-processing language to inject rules without restarting jobs. Deep learning (LSTM/CNN) trained on ~60–65 DGA families drops domain-detection false positives from double-digit rates to 0.7% false positive rate, after a ~9-month POC and roughly a year of tuning with Apache Flink and Kafka.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:05Siemens Cyber Defence Centre
- 2:25Why build a new detection platform
- 9:21Flink
- 11:26Why Flink
- 14:50Event Processing Language
- 15:57Aggregation Rule
- 18:01Attack Time Lag
- 20:50Not that easy
- 21:30Legal domains
- 21:45Different approaches
- 26:14How it works
- 28:28Platform
- 31:08Machine Learning
- 32:17Unsupervised Learning
- 38:13New Use Cases
related talks
25mReal-Time Fraud Detection at Scale
29mBuilding Resilient AI Platforms: Financial Fraud Detection
27mDataEngBytes 2023 - MEL-T1-08 - Troy Sellers
45mApache Flink — Real-Time Data Processing
53mData Mesh in Practice - Assuring Data Quality at Scale
16mAnalysing Real time Activity Streams for Security
44mBest practices for analytics and generative AI on AWS(ANT302)
13mBuilding Financial Fortresses: Real-Time Fraud Detection