Managing Multi-Cloud Apache Spark on Kubernetes
Talkat Kubernetes AI Day Europe 2022 · on Monday, 2022-05-16· at 16:00· Pavilion 3, Room D | Level 2 | Central Forum·
summary
Bloomberg runs Apache Spark across bare metal, EKS and AKS by moving executor creation out of user pods into an admin controller that reconciles a SparkApp custom resource, avoiding elevated user service accounts. Cluster Autoscaler events and pod termination reasons (OOM, preemption, scale‑down) are mirrored into a lightweight pod-status controller and per-app CR status so GUIs and notebooks see autoscale and failure causes. Finalizers are considered but avoided; deployment scales reliably to ~400–500 executors.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:53Background (Kubernetes)
- 1:27Background (Apache Spark)
- 2:10Background (Spark)
- 5:42User Stories (Complications)
- 6:27User Stories (Solutions)
- 9:57Why custom resource (CR)
- 10:44Storing information, where to?
- 12:32First: Cluster scaling up
- 13:52Cluster autoscaler events
- 15:08Controller to look up event objects
- 16:46Next: Scaling down, OOM, etc.
- 17:40Keeping pods?
- 19:25Kubernetes custom resource (CR)
- 20:18PodStatus Controller behavior
- 22:38Extra: Declarative copying
- 25:25Extensions
related talks
21mSpark on Kubernetes: The Elastic Story
29mDynamic Large Scale Spark on Kubernetes
29mScaling Apache Spark on Kube to Apple Scale
30mBeyond Experimental: Spark on Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
40mApache Spark on Kubernetes: A Technical Deep Dive
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
28mCloud-Native Apache Spark: why & how to migrate your Spark