Running Apache Flink everywhere
summary
Claims a reworked process model called Flink Improvement Proposal 6 that splits Flink into resource manager, dispatcher, job manager, task manager so deployments behave correctly on Standalone, YARN, Mesos, Docker/Kubernetes. Describes an Alibaba prototype that works: per-job job managers, one-shot client submits, dynamic slot allocation, resource profiles for heterogeneous containers, simpler classloading and stronger credential isolation. Sessions let multiple short jobs reuse warmed resources but carry credential risk; dispatcher exposes a REST entry point for multi-job clusters.
AI-written summary · about
chapters (20)
- 0:00Introduction to Flink deployments
- 1:05Current Flink deployment methods
- 3:56Job vs session clusters
- 5:22Resource management paradigms
- 9:13Flink Improvement Proposal 6
- 10:57Current Flink standalone model
- 12:01Flink on YARN architecture
- 13:23New component model
- 16:14Rebuilt Flink on YARN
- 17:17Benefits of new model
- 19:46Dispatcher for multi-job sessions
- 20:29Flink on Mesos
- 21:11Standalone mode evolution
- 22:14Flink on Kubernetes
- 23:17Container resource manager
- 25:23Auto-scaling with Kubernetes
- 26:27Multi-job sessions explained
- 29:59Dynamic resource handling
- 31:43Building blocks approach
- 32:45Q&A session
related talks
30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?
42mManaging Apache Flink on Kubernetes - FlinkK8sOperator
42mApache Flink deep-dive #bbuzz
43mApache Flink Meets Apache Mesos and DC/OS
38mApache Flink: A Very Quick Guide
25mData: Moving with the flow
42mStream Processing beyond Streaming Data
47mBring Flink to Large Scale Production