Bring Flink to Large Scale Production
summary
Alibaba runs Flink at 100 million events/sec peak using RocksDB state backend and introduces async external I/O to avoid blocking, yielding 2–3x throughput on many jobs. Checkpointing moves to incremental checkpoints (reuse SST files) to cut snapshot time and speed recovery; deployment shifts to per-job JobMaster/resource manager model for isolation and elasticity. Online streaming features, a StreamCompute SQL platform, and online RL deliver a reported 30% boost in conversion for search.
AI-written summary · about
chapters (18)
- 0:00Introduction to Flink at Alibaba
- 2:28Real-time data challenges at scale
- 7:28Twin streaming use cases
- 12:11Changing data and queries
- 15:29Why Flink fits these challenges
- 16:33Blink project for large-scale scaling
- 17:36Async I/O for external services
- 20:29State management in Flink
- 23:02State backends and checkpointing
- 25:59Flink 1.6 deployment improvements
- 28:53Monitoring and metrics in production
- 30:42Productivity challenges with DataStream API
- 32:06Introducing SQL for streaming
- 38:43Streaming SQL improvements
- 40:33Stream computer platform
- 43:01JobGraph approach for simplicity
- 44:51Architecture summary
- 46:18Takeaways and conclusion
related talks
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
21mFlink's application at Didi
25mData: Moving with the flow
45mApache Flink — Real-Time Data Processing
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
43mLeverage the power and simplicity of SQL on Apache Flink
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
38mApache Flink: A Very Quick Guide