Apache Spark and ML workflows
summary
Shows how to train churn models from a telco CSV using Spark MLlib pipelines (string indexer, one‑hot encoding, VectorAssembler) and how to export/apply models with PMML or serialized Python objects, warning that pickles are insecure and slow. Demonstrates distributed grid search by swapping scikit‑learn’s GridSearchCV for spark_sklearn so many hyperparameter fits run across the cluster instead of via n_jobs, and recommends using JPML/OpenScoring for safe distributed scoring in Java to avoid REST‑based DDoS on a single server.
AI-written summary · about
chapters (13)
- 0:00Intro
- 1:45STITCH FIX
- 10:26Modeling Life Cycle Persisted Model Training
- 14:42Churn Prediction Dataset
- 17:21Modeling Training
- 19:03Scikit-Learn Models
- 20:25Scikit-Learn Pipelines
- 21:35Apache Spark MLlib Pipelines
- 28:51Storing Models as PMML
- 31:08Model Life Cycle
- 33:16Embarrassingly Parallel Model Training
- 39:01What do you mean deploy?
- 40:46Scoring with a REST server: Open Scoring
related talks
39mMachine Learning with Scala on Spark
50mApache Spark vs Machine Learning
23mRethinking scalable machine learning with Spark ecosystem
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
31mFrom Pandas to Spark
35mPySpark : Combining Machine Learning & Big Data
55mGetting Started with Spark and ML
17mMachine Learning at the Speed of Light with Spark