A streaming correlator handling over 2M EPS

on YouTube ↗41m

" summary

Describes a streaming correlator built to process over 2 million events per second (EPS) for modern Security Operation Centers handling hundreds of systems and tens of gigabytes per second. Covers scalability challenges of SIEMs in BigTech-scale environments and presents a high-throughput, horizontally scalable streaming approach to correlate and process massive security event streams.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 1:30Old system problems
3. 10:27What is Flink
4. 17:30First correlator version
5. 24:51Current architecture
6. 30:08Results and issues
7. 38:54Q&A

" related talks

• 39mLarge scale streaming analyticsJay Palaniappan · Data Con LA 2019
• 50mHow We Store 60,000 Events per SecondArsen Mukuchyan · HighLoad++ 2013
• 25mThe Importance Of Real time Data In Fighting Cyber AttacksLior Mozel, Eviatar Tenne · Aerospike Digital Summit 2021: Reimagine Real-time
• 19mScale observability and secure data in motion with Elastic and ConfluentJennifer Garcia, Anand Venugopal · Microsoft Build 2023
• 01h 05mPractical network defense at scaleTravis Carelock · CODE BLUE 2015
• 39mAdvanced Stream Processing on the EdgeEduardo Silva · linux.conf.au 2020
• 31mHelp I need a stream processor - learning how to chose ...Andrew Psaltis · Berlin Buzzwords 2016
• 38mStreaming analytics and CEP ... #bbuzzTill Rohrmann, Fabian Hueske · Berlin Buzzwords 2016