Scaling Apache Spark on Kube to Apple Scale
summary
Apple migrates large Spark workloads from YARN/Mesos to Kubernetes and reports concrete results: small-to-medium ETL jobs run well after increasing resource requests, while long ETLs suffer overcommit and complex ML and streaming jobs fail due to expensive recomputation and checkpointing gaps. Engineers implement graceful decommissioning, add dynamic scaling and external shuffle storage, integrate PVCs and explicit shuffle-file removal, and find caching and ephemeral disk matter; these changes work but need tuning and better documentation.
AI-written summary · about
chapters (21)
- 0:00Start
- 0:25Why You Should Put Spark on Kubernetes
- 1:39What Is Spark
- 2:50Spark Is a Good Way for Folks To Learn Functional Programming
- 3:21How Spark Works
- 5:12What Is Yarn
- 6:22What Is Mesos
- 7:16Standalone Spark
- 7:43Why We Should Put Spark on Kubernetes
- 11:27Containers and Python and Security
- 13:45What Worked Well
- 14:13Challenges around Integration with the Different Data Sources
- 15:09Dependency Management
- 19:03Increase Your Resource Requests
- 21:45Adding a Truly External Shuffle Service
- 23:07External Shuffle Storage
- 23:50What Were some Areas for Improvement
- 25:01Avoiding Cascading Failures
- 26:46Dynamic Preemption Priorities
- 27:48Handling of Jobs with Changing Priorities or Deadlines
- 28:21Conclusion
related talks
21mSpark on Kubernetes: The Elastic Story
31mManaging Multi-Cloud Apache Spark on Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
41mRunning Apache Spark on K8s: From AWS EMR to K8s
29mDynamic Large Scale Spark on Kubernetes
28mCloud-Native Apache Spark: why & how to migrate your Spark