Integrate Flink with Kubernetes natively

on YouTube ↗37m

" summary

Flink embeds a KubernetesResourceManager that calls the Kubernetes API to create and release TaskManager pods on demand, supports both per-job and session cluster modes (per-job spins up a dedicated cluster and destroys it on completion), and demonstrates dynamic scaling and automatic TM release in the demo. High availability uses HDFS checkpoints plus leader election via ConfigMap leader election, which recovers jobs after killing a JobManager. Logging options: custom distributed appender, sidecar or DaemonSet; feature ships as a preview in 1.10 with fuller application-mode work planned for 1.11.

AI-written summary · about

" chapters (17)

1. 0:00Introduction
2. 0:30Agenda
3. 1:09What is Kubernetes
4. 1:46Architecture of Kubernetes
5. 3:12Concepts of Kubernetes
6. 6:51Why run Flink on quantities
7. 8:40Flink and Kubernetes together
8. 9:12Isolation
9. 10:27Operators
10. 11:39Limitations
11. 14:12Technical bound
12. 18:52Job Master
13. 19:35Station vs Job
14. 21:04Demos
15. 28:03Optimizations
16. 30:15Locks and Matrix
17. 32:39Network

" related talks

• 37mStateful Stream Processing with Apache Flink 1.5 and beyondNico Kruber · Berlin Buzzwords 2018
• 33mFlink in ContainerlandPatrick Lucas · Flink Forward Berlin 2017
• 42mManaging Apache Flink on Kubernetes - FlinkK8sOperatorAnand Swaminathan · KubeCon + CloudNativeCon North America 2019
• 25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8sMichas Szacillo, Wang Li · KubeCon + CloudNativeCon Europe 2025
• 20mAutoscaling Apache Flink ApplicationsRobert Metzger · Berlin Buzzwords 2021
• 37mSpark and Flink Running Scalable in Kubernetes #bbuzzFrank Conrad · Berlin Buzzwords 2018
• 44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & BeyondNaci Simsek · Big Data Conference Europe 2025
• 26mKubernetes & real-time analytics: How to connect these worlds with Apache FlinkAlbert Lewandowski · Berlin Buzzwords 2021