Distributed Computing in Clojure and Spark
summary
Shows how to write Clojure code that runs on Apache Spark using the Sparkling interop, package it as an uberjar, and submit it to a Spark cluster running on Kubernetes. Demonstrates a real example on Brooklyn real-estate CSVs computing price per square foot and finding Brighton Beach highest. Covers REPL-driven development, using gen-class, handling Spark/Hadoop dependency version mismatches, recommending minikube with ~8 GB for local testing, and reports that deployments work after resolving subdependency conflicts.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 1:08When to use Spark
- 4:48Spark ecosystem overview
- 6:56Spark RDD fundamentals
- 11:23Brooklyn real estate demo
- 16:24Local mode and development
- 18:10Deployment challenges
- 20:00Cluster provisioning options
- 21:51Deploying Spark jobs
- 25:42Q&A and resources
related talks
43mDoing Data Science with Clojure
38mMachine Learning with Clojure and Apache Spark
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
37mClojure Remixed: Pyroclast Data Processing
31mLightning Talks
37mCascalog 2- Streaming MapReduce in Clojure
31mManaging Multi-Cloud Apache Spark on Kubernetes
34mClojure by neophyte