The Future of Apache Flink
summary
Current Flink road map lists working features and rough designs: production-ready Dynamic Scaling lets jobs change parallelism at runtime, a Window Trigger DSL (in master via PR) replaces hundreds of lines of custom trigger code, and a prototype Queryable State exposes internal state for external reads. In-progress items include side inputs/outputs, window firing context, incremental checkpointing, and hot standby for replicated state; longer-term work covers streaming SQL, native Mesos/YARN/Docker integration, security hardening, and failure policies with live operator inspection.
AI-written summary · about
chapters (35)
- 0:00Intro
- 1:30Disclaimer
- 2:07Topics
- 3:07Features
- 4:01Streaming API
- 7:52Triggers
- 8:48Window Trigger DSL
- 10:27Window Function
- 12:17Window Operator
- 13:56Queryable State
- 14:38Stream Processor as a Database
- 15:14Window Operations
- 15:52Side Inputs
- 16:54Late Data
- 17:17Slate Data
- 17:36Side Outputs
- 18:24Stream SQL
- 19:26State Checkpointing
- 19:36Job Checkpointing
- 21:03Incremental Checkpointing
- 21:50Hot Standby
- 23:04Why do we need these
- 23:42Operations
- 24:19Big Data
- 24:32Data Spike
- 24:54Dynamic Scaling
- 25:39Cluster Management
- 26:23Flink Everywhere
- 26:50Cluster SSD City
- 27:17Dynamic Resource Requests
- 27:38Security Enhancements
- 28:29Failure Policies
- 29:40Flink Improvement Proposals
- 30:36Recap
- 32:16Questions
related talks
38mApache Flink: A Very Quick Guide
43mLeverage the power and simplicity of SQL on Apache Flink
42mApache Flink deep-dive #bbuzz
41mBeyond the Hype: When Apache Flink Solves Real Problems
37mA Year in Flink: The Most Important Changes in Recent Versions
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
42mStream Processing beyond Streaming Data
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?