Apache Flink — Real-Time Data Processing
summary
Apache Flink is used to process 200,000 requests/sec at DataDome, evaluating about 300,000 behavioral rules to decide blocking with P99 latency around 100 ms (median single-digit ms). The pipeline uses Kafka sources, five Flink jobs across ~200 task servers (roughly 100 for behavior), stores state in RAM per slot (~50 GB/machine) or RocksDB with checkpoints persisted to S3, and writes outputs to Kafka/S3; it achieves linear scalability and low latency but struggles with limited elasticity, difficult stateful upgrades, and operates with practical at-least-once semantics.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:05Datadome use cases
- 5:01Deep dive Flink job
- 14:13Cluster deployment
- 20:51State management
- 33:47Tool comparison
- 42:10Demo and conclusion
related talks
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
43mLarge Scale User Behavior Analytics by Flink
42mApache Flink deep-dive #bbuzz
25mData: Moving with the flow
38mApache Flink: A Very Quick Guide
34mA journey of scaling high-volume low-latency stateful processing
41mBeyond the Hype: When Apache Flink Solves Real Problems
47mBring Flink to Large Scale Production