Spam-fighting Infrastructure
summary
Describes an executable rule engine built on Haxl rule engine that enforces pure-function rules with lazy, overlapping data fetches to keep synchronous path latency tiny; explains streaming counters like DGIM and sliding HyperLogLog for approximate time-window counts and uniques, a batching update flow into a point-query KV store with lazy deletion and no retries, and a clustering pipeline that trains locality sensitive hashing on feature vectors, matches against a fast index, and K‑means clusters windows to catch large repetitive campaigns.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:53Problem looks simple
- 1:09Problem solved!
- 2:34We have a log?
- 4:06What are rules?
- 4:17Lazy is good ;-
- 5:00Latency matters
- 5:30Move fast and NOT break things!
- 6:29All we need
- 7:37Data Time is the key
- 8:39Example (previous talk)
- 8:55Plain data Nope, too slow
- 9:47Preaggregate Everything! Compute your stats in advance
- 10:23What do we want?
- 11:25Solution: streaming Data structures
- 12:47DGIM
- 16:48The system
- 17:27Key decisions
- 18:55Advanced adversary Abandon all hope
- 20:28Clustering photos, text, requests, code
- 21:14The Power of clustering Connect unrelated things (visualization)
- 21:55The power cd... Content is expensive
- 23:02Our solution Unified representation Two subproblems: Locality sensitive hashing
- 24:18Example Nilsimsa hash
- 26:09Generic framework LSH model
- 27:57Clustering pipeline
- 28:51Online clustering Sliding time window
- 29:37Tradeoffs
- 31:20Summary
related talks
45m2014.10.29 - Security@Scale - Louis Brandy, Facebook
38mNext-Gen Detection: Harnessing LLMs for Sigma Rule Automation
41mHow to Elm-ify Your ML
38mPhishing LLMs: Reeling in the Machine
23mFighting phishing and spam with online machine learning on data streams
45mSweeping Up...
40mLeveraging Streaming-Based Outlier Detection and SliceLine to Stop Heavily Distributed Bot Attacks
24mCultivating an Engineering Dialect