Apache Spark on Kubernetes: A Technical Deep Dive
summary
Kubernetes becomes a native Spark scheduler in Spark 2.3 with a submission client that creates a driver pod plus a scheduler backend running in the driver to manage executor pods; the system uses a headless Service for driver‑executor networking and supports container or remote dependency staging. Spark 2.4 removes the init downloader in favor of running the driver container for dependency fetch, adds fractional CPU support, introduces an operator CRD with cron and Prometheus integration, and plans Kerberos support and pod templates and better dynamic allocation. Data locality remains unimplemented.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:29Why Spark on Kubernetes?
- 4:04Apache Spark
- 5:22Brief Project History
- 7:10Spark Meets Kubernetes
- 7:41Kubernetes Submission Client
- 8:57Kubernetes Scheduler Backend
- 10:52Under the Hood
- 13:27Executors Connecting to Driver
- 15:06Dependency Management
- 19:00Existing Features
- 21:15Client Mode (2.4)
- 23:53Kerberos Support (3.0)
- 25:46Kubernetes Operator for Spark
- 28:11Roadmap (3.0 and Beyond)
- 31:31Getting involved
related talks
01mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
29mDynamic Large Scale Spark on Kubernetes
28mCloud-Native Apache Spark: why & how to migrate your Spark
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
21mSpark on Kubernetes: The Elastic Story
30mBeyond Experimental: Spark on Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
31mManaging Multi-Cloud Apache Spark on Kubernetes