SPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
summary
Built an elastic Spark-on-Kubernetes platform that moves workloads from EMR/YARN to Kubernetes using the Spark Operator, Argo CD/workflows, Jupyter on K8s and S3, with custom S3a libraries and a Spark History Server. Claims 25-30% cost savings and deploy time down from 7+ days to <1 day. Demonstrates daily XGBoost batch inference and reusable feature pipelines, solves Spark 2.5 quirks (dynamic allocation, overwritten spark.properties) with configmap merging, and recommends Spark 3.0.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:08Containers vs Bootstraps
- 3:54CD for deployment
- 6:50Example: QuickBooks Mobile Push notification model
- 8:39Daily batch inference using EMR
- 9:38Daily batch inference using Spok
- 11:03Spok can be invoked from notebooks
- 12:26Example: QuickBooks Online transaction categorization
- 13:53Feature ETL logic is a simple JOIN
- 15:24Transaction Categorization Spark Application in ArgoCD
- 17:39Spark 3 is best for Kubernetes
- 18:30Spark Operator mounts over
- 19:24Summary: Advantages of Spok over YARN
related talks
29mDynamic Large Scale Spark on Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
21mSpark on Kubernetes: The Elastic Story
41mRunning Apache Spark on K8s: From AWS EMR to K8s
30mBeyond Experimental: Spark on Kubernetes