Real-time bot mitigation with machine learning in Flink
summary
Datadome runs real-time bot blocking on Apache Flink over ~5 billion requests/day and produces per-IP verdicts with a business target of <0.01% false positives. They build Sherlock (detection) plus Watson (auto-thresholding) that clusters per-customer measures with a density-based clustering pass and outputs auto-rules injected back into Sherlock. They handle 200+ customers with a custom hybrid window (min time + min points), cap threshold swings to 10× to avoid outages, and fix severe backpressure by switching from value state to list state. The system runs in production and continuously adapts thresholds.
AI-written summary · about
chapters (4)
related talks
25mReal-Time Fraud Detection at Scale
40mLeveraging Streaming-Based Outlier Detection and SliceLine to Stop Heavily Distributed Bot Attacks
13mBuilding Financial Fortresses: Real-Time Fraud Detection
29mBuilding Resilient AI Platforms: Financial Fraud Detection
15mData science at scale using Apache Flink: Dynamic model serving and real-time feature generation
10mReal-Time Fraud Prevention at Scale
27mDataEngBytes 2023 - MEL-T1-08 - Troy Sellers
47mAI vs. Karen: Real-Time Customer Sentiment Detection at Scale