Machine Learning Workflows with HashiCorp Nomad and Apache Spark
summary
HashiCorp provides an open-source Nomad-Spark integration that runs Apache Spark executors and drivers as Nomad tasks, supporting both cluster mode and client mode and passing all spark-submit options through to Nomad. Nomad schedules executors with templates that allow custom resource claims (including device plugins for NVIDIA GPUs), dynamic allocation, job priority and multi-region targeting. The system is proven at scale—Nomad clusters exceed 10,000 nodes and achieve thousands of job deployments per second—used successfully in production.
AI-written summary · about
chapters (6)
- 0:00Introduction to talk
- 1:25What is HashiCorp Nomad
- 8:43Apache Spark overview
- 11:53Why Nomad for Spark
- 13:40Running Spark on Nomad
- 21:05Summary and resources
related talks
01h 06mSchedule AI workloads efficiently on Nomad with node pools
53mBootiful workload orchestration with Hashicorp Nomad
36mAutomating Cloud-native Spark Jobs with Argo Workflows
12mNomad: Lightweight Orchestration That Complements Kubernetes
31mOrchestration with Nomad
30mAutoscaling with HashiCorp Nomad
28mCloud-Native Apache Spark: why & how to migrate your Spark
29mDynamic Large Scale Spark on Kubernetes