Scaling and Securing Spark on Kubernetes at Bloomberg
summary
Bloomberg runs Spark natively on Kubernetes with a working External Shuffle Service plugin API (SPIP spark 25299) that lets shuffle bytes be stored on external file servers or DFS to enable dynamic resource allocation and disaggregated compute. A privileged controller uses an exact pod scaler CRD to atomically provision drivers and executors under per-namespace quotas, avoiding hung drivers. A centralized token renewal service issues and rotates Kerberos/Hadoop delegation tokens via secrets and a Hadoop UGI patch so long-running jobs keep access to HDFS/S3.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:39Data Science Platform
- 1:36Primary ETL component: Apache Spark
- 3:31Colocated vs. Disaggregated Compute
- 5:14Example Job CRD Lifecycle
- 5:56Spark Standalone on kubernetes
- 7:33Spark Native on Kubernetes
- 9:07Increased Security
- 10:13Secure Executor Creation
- 12:07Secure Data Communication
- 15:35Spark Shuffle
- 18:04Spark Temporary Files
- 18:46Current state of the External Shuffle Service
- 21:27API for Pluggable Remote Storage
- 22:28Individual File Servers
- 24:53Asynchronous Backup
- 26:06Direct writes to DFS
- 33:39Edge Cases and Further Extensions
related talks
43mSpark Over a Mesos Secured Productive Environment
23mApache Spark meets FIPS standard
29mScaling to Millions of ML Models to Solve the Problems of SRE and Security
40mArchitecting Secure Agentic Workflows on Kubernetes: A Financial Sector...
23mObtain, Distribute and Use Temporary Credentials Automatically in Apache Spark and Apache Flink
31mRings of Security for Big Data in Cloud
25mSecuring AI Pipelines: Real-World Attacks on Kubernetes-Based AI Infrastructure
32mKubernetes Namespaces Unleashed: Unlocking the Full Potential of You...