Flink in Containerland
summary
Built and published an official Flink Docker image and runs Flink on Kubernetes, but networking and addressability cause the biggest failures: Akka-based RPC drops external submissions unless they come from inside the cluster (Akka RPC addressability). Solutions used include pinning blob and queryable-state ports, running job manager in HA with ZooKeeper, building fat jars or custom images for extra libs, and submitting jobs via the REST/UI as a workaround (HTTP API for job submission). Recommends ~1 CPU per slot, ~1–4 GB RAM per task manager and a proper HTTP management API.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 3:03Flink and K8s basics
- 10:08Networking and RPC
- 19:17State and Security
- 26:07Future improvements
- 28:10Q&A session
related talks
15mFlink's New Dockerfile: One File to Rule Them All
44mLets Deeploy Flink - Uncovering Hidden Depths of Yarn, Docker, Kubernetes & Beyond
37mIntegrate Flink with Kubernetes natively
22mBuilding a Kubernetes Operator for Apache Flink in Java
42mManaging Apache Flink on Kubernetes - FlinkK8sOperator
25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8s
37mSpark and Flink Running Scalable in Kubernetes #bbuzz
20mFlink powers Houzz near-realtime pipeline & Kubernetes logging observability