To Normalized Logs, and Beyond - Building a Threat Detection...
summary
Built a replaceable, serverless AWS pipeline using Lambda, EventBridge, SQS, Kinesis Firehose and S3 to centralize heterogeneous logs, normalize with a lightweight wrapper schema while retaining raw messages, and enrich via DynamoDB-to-CSV for a commercial detection engine; detections run as code with dev/prod separation, unit tests and continuous validation canaries that fire synthetic events and CloudWatch alarms. Forensics automation snapshots EBS, mounts in a prepared AMI, and Terraform modules speed onboarding. Current scale hits tens of thousands of events/sec with plans for millions of events/sec.
AI-written summary · about
chapters (16)
- 0:00Start
- 5:07Detections & Alerts
- 9:19Data Collection
- 22:45High-Level Architecture
- 23:13What is a Detection?
- 24:47How do we manage detections?
- 25:56Detection Workflow
- 28:55Detection Health System
- 30:09Component 1-Generating Canary Alerts
- 34:01Basic Enrichment Example
- 36:00Enrichment Architecture
- 37:31Investigation
- 38:53Fail to prepare, prepare to fail
- 42:13Detection Lifecycle
- 43:14Learnings
- 45:00Alternative Approaches
related talks
28mPatterns for scalable security detections on AWS
32mBuilding and scaling a log analytics platform - a serverless approach
01h 02mSecuring data and democratizing the alert landscape
01h 02mCentralised Logging without the blood, sweat and tears
25mLogging, Monitoring, and Alerting in AWS (The TL;DR)
37mA Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day
28mPatterns for scalable security detections on AWS
55mStreamline security analysis with Amazon Detective