Apache Spark vs Machine Learning
summary
Compares Apache Spark’s ML capabilities using the PySpark DataFrame API (lower/replace/split/explode, window functions, and Catalyst optimizations). Demonstrates training a RandomForest in a Spark ML pipeline (70/30 split, maxDepth=8, minInstancesPerNode=5), persisting/reloading the model, and achieving ~77% accuracy on a 24-class letters dataset. Shows that the tokenization/counting pipeline runs interactively and that model training and deployment scale from a single node up to an 8000-node cluster.
AI-written summary · about
chapters (8)
- 0:00Spark & ML intro
- 11:28Spark architecture
- 17:31Working with Spark data
- 28:11Spark scalability
- 32:29ML models in Spark
- 39:52Building Spark ML models
- 45:39Summary: Spark vs ML
- 46:22Q&A
related talks
17mMachine Learning at the Speed of Light with Spark
39mMachine Learning with Scala on Spark
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
35mPySpark : Combining Machine Learning & Big Data
51mApache Spark and machine learning: dreams and realities
44mData Science at Scale with Apache Spark
44mApache Spark and ML workflows
48mMachine Learning in Azure Databricks