From Pandas to Spark
summary
How to adapt ML models from single-machine Python tools to distributed frameworks: replacing Gensim Word2Vec with PySpark MLlib equivalents, comparing truncated SVD implementations in PySpark MLlib and scikit-learn, and approaches to distributed neural-network training (Keras/TensorFlow) on HDInsight clusters.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:04Project lifecycle
- 2:26Case task and context
- 3:28Choosing Spark
- 5:11Current pipeline
- 6:58SVD decomposition issues
- 9:11Word2Vec issues
- 11:18Spark MLlib limitations
- 12:23Word2Vec replacement
- 14:34Word2Vec implementations comparison
- 16:25Distributed SVD
- 18:53Training on samples issues
- 19:56Distributed neural networks
- 22:29Summary
- 23:36Q&A
related talks
35mPySpark : Combining Machine Learning & Big Data
50mApache Spark vs Machine Learning
44mApache Spark and ML workflows
23mRethinking scalable machine learning with Spark ecosystem
39mMachine Learning with Scala on Spark
17mMachine Learning at the Speed of Light with Spark
27mMachine Learning at Scale
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons