SAQL: A Stream-based Query System for Real-Time Abnormal System Behavior Detection
summary
Built a stream query engine called SAQL with a domain-specific language for rule-, time-series-, invariant-, and outlier-based anomalies, using stateful sliding-window aggregation and a master–dependent query sharing scheme over a CEP backend (CD in transcript). Deployed at NEC Labs America over 150 hosts and evaluated on 1.1 TB / 3.3B events; reports 2s detection latency and 110,000 events/s throughput (~4,000 hosts equivalent) with lower memory use than general CEP systems.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:38The Equifax Data Breach
- 0:53Impact of Advanced Persistent Threat (APT) Attack
- 1:22APT Attack: Case Study
- 3:32Ubiquitous System Monitoring
- 4:19Challenge 1. Attack Behavior Specification
- 4:41Challenge 2. Timely "Big Data" Security Analysis
- 6:15Data Collection
- 6:31Rule-based Anomaly: Single-Event
- 8:12Time-Series Anomaly
- 9:12Invariant-based Anomaly
- 10:09Outlier-based Anomaly
- 10:52SAQL Execution Engine
- 11:32Master-Dependent-Query Scheme
- 12:57Case Study: Four Major Types of Attacks
- 13:54Case Study: Execution Statistics
- 14:24Pressure Test
- 15:33Performance of Concurrent Query Execution
- 17:53Alert Detection and Investigation
- 19:37Conclusion
related talks
39mTransform data investigation via Elasticsearch Query Language (ES|QL)
38mStreaming analytics and CEP ... #bbuzz
27mLarge Scale Stream Processing with Blink SQL at Alibaba
42mdistributed stream processing for everyone with KSQL #bbuzz
24mThreat Hunting Powered by Efficient and Straightforward Anomaly Detection on Your Data Lake
39mDeclarative stream processing...
23mQuery like a Pro: ES|QL meets Apache Arrow and Pandas
45mKafka and KSQL to help with detect...