Real-time Experiment Analytics at Pinterest with Apache Flink
summary
Real-time pipeline built on Apache Flink computes experiment metrics and runs statistical tests every 15 minutes, cutting metric latency from 24 hours to 15 minutes and accumulating up to three days of data for stronger tests. It keys joins activation and event streams, uses low-level APIs like KeyedProcessFunction with a map state of timestamp→(num_actions,num_users), handles out-of-orderness via multi-slot updates and a 15-minute tumbling emit, copes with 150 GB state and millions of messages/sec by capping per-user spam and diagnosing backpressure causing checkpoint stalls.
AI-written summary · about
chapters (7)
- 0:00Intro
- 2:16Commonly used terms
- 6:17Realtime Dashboard Overview
- 6:58Scale of the problem
- 9:09Input message format
- 13:26Checkpoint failure
- 16:07Data Validation
related talks
25mSharing what we love: The Apache Flink story at Pinterest
45mApache Flink — Real-Time Data Processing
43mLarge Scale User Behavior Analytics by Flink
27mArchitecting Sub-Second Analytics Pipelines
57mEmergence of Real-Time Analytics: Real-time Analysis of...
43mA Data Streaming Architecture with Apache Flink #bbuzz
28mMigration of a realtime stats product from Storm to Flink
40mRobust Stream Processing with Apache Flink