Spark on Kubernetes: The Elastic Story
summary
Apple runs Apache Spark on Kubernetes using the Spark Kubernetes Operator, per-queue unicorn resource queues and Kubernetes Cluster Autoscaler to build a cloud-native Spark service with reactive autoscaling and per-queue node groups. Physical isolation maps drivers and executors to dedicated node groups, enables bin-packing, and uses EC2 scale from 2–200 nodes; scale-out latency stays under five minutes. Production usage runs hundreds of thousands of vCPUs and hundreds of TB RAM, cutting queue costs by up to 70–80% and averaging 20–70% savings.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:10Bowen Li Apple
- 1:06Outline
- 1:43Why Cloud
- 3:23Design Principles
- 5:59Elastic Spark Architecture
- 10:33Open Source
- 12:11Node Groups Layout
- 14:46Cluster Autoscaling
- 16:57Production Status
- 18:36Key takeaways
- 19:51Whats next
related talks
30mBeyond Experimental: Spark on Kubernetes
29mScaling Apache Spark on Kube to Apple Scale
29mDynamic Large Scale Spark on Kubernetes
39mSpark on Kubernetes: Best Practice and Performance
31mManaging Multi-Cloud Apache Spark on Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
40mApache Spark on Kubernetes: A Technical Deep Dive
28mCloud-Native Apache Spark: why & how to migrate your Spark