Apache Flink 2.0: Unified Enterprise Data Processing System and Beyond
summary
Alibabas Singles Day runs on Flink and hits 2 billion events per second, processing ~3 TB/s and supporting 500k+ transactions/s while generating $38B sales in 24 hours. Flink claims pipeline execution, native event‑time semantics, and exactly‑once end‑to‑end when used with Kafka transactions and checkpoints. It uses local state with RocksDB state backend for terabytes of state, a unified planner (stream+batch) since 1.9, ML pipelines with embedded TensorFlow operators for online training and inference, and a new serverless glue called StateFun.
AI-written summary · about
chapters (36)
- 0:00Introduction
- 0:24Agenda
- 1:50Singles Day
- 2:24Sales Data
- 3:22Online Machine Learning
- 4:20RealTime GMB Dashboard
- 4:44RealTime GMB Architecture
- 5:12Companies Leveraging Flink
- 6:40Typical Pipeline
- 7:15State
- 7:50State Control
- 8:21State Backend
- 9:12Flink Philosophy
- 10:12Pipeline Execution
- 11:11Processing Time
- 12:33Layered API
- 13:26Scalable
- 13:48Operational Friendly
- 14:12Recovery
- 14:25Exactly Once
- 16:01Recap
- 16:20Unified Enterprise Data Processing
- 18:40Flink Stack
- 20:23Flink Integration
- 21:04Flink Query
- 21:27Lambda Architecture
- 21:56Flink Architecture
- 24:04Machine Learning Library
- 24:24Machine Learning Pipeline
- 25:10Deep Learning Pipeline
- 26:12Solution
- 26:58Service
- 28:14Summary
- 28:38State Function
- 29:12Conclusion
- 29:38Flink vs Spark
related talks
45mApache Flink — Real-Time Data Processing
47mBring Flink to Large Scale Production
42mApache Flink deep-dive #bbuzz
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
38mApache Flink: A Very Quick Guide
43mLeverage the power and simplicity of SQL on Apache Flink
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
25mData: Moving with the flow