Optimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.
summary
Prototype builds a near‑real‑time cycling routing system that fuses traffic and air‑quality feeds, runs Spark on Kubernetes with Spark Operator, uses Kafka for ingestion, RocksDB for state, and a mix of CPU and GPU (Nvidia Rapids) for training. Shuffle causes data loss on executor failure so an external shuffle (IBM tool, AWS‑only so far) is required; dynamic allocation remains brittle. Monitoring is done with Prometheus. End‑to‑end works at human‑scale (55‑minute route example) but lacks large‑scale production deployment.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 3:24Real-time ML fundamentals
- 10:27Challenges in real-time ML
- 13:16Spark on Kubernetes benefits
- 15:25Key challenges running Spark on K8s
- 18:42Streaming and stateful processing
- 21:12Latency and configuration
- 22:37ML training on Spark
- 23:39Custom scheduling options
- 24:01Deployment and optimization
- 24:43Conclusion and open source contribution
related talks
29mDynamic Large Scale Spark on Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
26mRe-architecting the Data Platform with Spark on Kubernetes
21mSpark on Kubernetes: The Elastic Story
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
39mSpark on Kubernetes: Best Practice and Performance