Spark on Kubernetes: Best Practice and Performance
summary
Tencent describes running Spark on Kubernetes (Spark 2.3/2.4) as a production service with namespace/quota multi-tenancy, custom authorization and logging, and Zookeeper for driver high availability. Performance tuning covers both Spark and Kubernetes settings, mounting local volumes to avoid shuffle penalties, and limits such as no data locality on current Spark-on-K8s. Benchmark uses TPC-DS 1TB and shows full-run times around three hours when storage is separated, with measurable slowdowns from pod scheduling and data placement.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:23What is Spark
- 9:17Why do we need Kubernetes
- 13:46Architecture
- 14:16Spark Application
- 17:06Spark on accumulated status
- 21:50Applications
- 23:02Storage
- 24:00Service
- 25:26Structure
- 26:10HDFS
- 27:44Catalog
- 28:13Highs
- 28:50Environments
- 30:25Benchmark Configuration
- 33:32Benchmark Results
- 34:47Data Locality
- 36:59Our Experience
- 37:34Summary
related talks
21mSpark on Kubernetes: The Elastic Story
29mDynamic Large Scale Spark on Kubernetes
46mStorage Requirements and Options for Running Spark on Kubernetes
28mCloud-Native Apache Spark: why & how to migrate your Spark
30mBeyond Experimental: Spark on Kubernetes
52mSpark in K8s for dozens of DS teams
26mBuilding Apache Spark Clusters on K8s for AI Workloads
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available