Flink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8s

on YouTube ↗25m

" summary

Bloomberg runs 1,000 Flink jobs across multiple Kubernetes clusters and builds a Karmada extension that preserves Flink state for automated cross‑cluster failover. The implementation adds state preservation via JSONPath in the propagation policy to copy jobID into labels, and a Kyverno mutating webhook injects the initial savepoint path so a redeployed job resumes from its checkpoint. It reduces manual redeploys and eases maintenance windows, but requires tuning tolerationSeconds and has some cluster‑failover gaps still being addressed.

AI-written summary · about

" chapters (22)

1. 0:00Flink on Karmada intro
2. 1:07Speakers and agenda
3. 2:09Bloomberg’s streaming platform
4. 3:14Flink fundamentals and state
5. 4:39Flink on K8s with Operator
6. 6:04Multi-cluster Flink challenges
7. 8:30Control plane for resilient pipelines
8. 9:12Karmada for multi-cluster
9. 10:14Karmada health tracking
10. 11:15First Flink-Karmada integration
11. 12:22Initial Karmada-Flink limits
12. 13:03Flink job state for health
13. 14:50Unhealthy state detection
14. 15:54Automated failover in Karmada
15. 16:36State preservation on failover
16. 18:21Failover workflow with state
17. 19:26Karmada benefits for Flink
18. 20:48Generalizing failover solution
19. 21:50Failover sensitivity config
20. 22:31Limitations and improvements
21. 23:53Future of failover and state
22. 24:14Community and next steps

" related talks

• 37mHow Bloomberg Creates a Resilient Data Analytics Platform Using KarmadaMichas Szacillo, Ilan Filonenko · KubeCon + CloudNativeCon China 2025
• 37mIntegrate Flink with Kubernetes nativelyYang Wang · Flink Forward Virtual 2020
• 16mZero-downtime upgrades of Flink applicationsLakshmi Rao · Flink Forward Global Virtual 2020
• 33mFlink in ContainerlandPatrick Lucas · Flink Forward Berlin 2017
• 42mManaging Apache Flink on Kubernetes - FlinkK8sOperatorAnand Swaminathan · KubeCon + CloudNativeCon North America 2019
• 20mFlink powers Houzz near-realtime pipeline & Kubernetes logging observabilityMenglei Sun · Flink Forward Global Virtual 2020
• 26mKubernetes & real-time analytics: How to connect these worlds with Apache FlinkAlbert Lewandowski · Berlin Buzzwords 2021
• 25mMulti-cluster Orchestration System: Karmada Updates and Use CasesHongcai Ren · KubeCon + CloudNativeCon China 2025