A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
summary
Compares a full pipeline run twice on the same messy dataset — exploration, cleaning, feature engineering, model fitting — showing that Spark.ml uses DataFrames not RDDs so Python performance equals Scala for many ops, that hyperparameter grid search parallelizes across a cluster and finishes surprisingly fast, and that pipelines persist, ETL and ML run on one platform; downsides include PySpark JVM overhead, awkward vector assembler and feature indexing, missing nonlinear kernels and limited native deep learning.
AI-written summary · about
chapters (17)
- 0:00Scala Spark vs Python Scikit-learn
- 2:07Scala Spark for machine learning
- 9:51Scala and Spark integration
- 13:17Spark distributed computing
- 16:03PySpark performance pitfalls
- 19:18Distributed data frames in Spark
- 21:03Spark vs Scikit-learn pipelines
- 23:30Spark replacing Hadoop
- 25:55Spark ML libraries
- 26:57ML on Spark cluster
- 29:26Scaling data science teams
- 31:11Python vs Scala Spark case studies
- 32:38Categorical features in Spark
- 34:04Graph processing and NLP in Spark
- 35:28Spark features and limits
- 36:34Scala Spark projects
- 37:39Q&A on nonlinear kernels
related talks
39mMachine Learning with Scala on Spark
50mApache Spark vs Machine Learning
23mRethinking scalable machine learning with Spark ecosystem
44mApache Spark and ML workflows
17mMachine Learning at the Speed of Light with Spark
42mScalable Data Science in Python and R on Apache Spark
35mPySpark : Combining Machine Learning & Big Data
31mFrom Pandas to Spark