Using Spark MLlib for NLP

on YouTube ↗34m

" summary

Built a standalone system called IdiML that turns text into feature vectors, trains logistic‑regression models with Spark MLlib vectors (not DataFrames), and serves thousands of low‑latency predictions per second. It packages a feature pipeline, model, metadata and versioning as a single JAR object, supports automated continuous training and quick rollback, and replaced a previous cluster of ~24 servers — matching that throughput on a consumer laptop by skipping DataFrame overhead and tuning vector‑level use of MLlib.

AI-written summary · about

" chapters (11)

1. 0:00Introduction to NLP with Spark
2. 1:19Why NLP matters in real world
3. 5:13Training data and model building
4. 6:39Feature extraction pipeline
5. 9:26Model training and prediction
6. 10:49Why EML is needed
7. 15:46Hyperparameter tuning
8. 18:03Version control in models
9. 23:20Performance optimization
10. 27:21Future improvements
11. 30:15Q&A session

" related talks

• 39mMachine Learning with Scala on SparkJose Quesada · Scala Days Berlin 2016
• 45mDeep Learning and NLP with SparkMelanie Warrick, Andy Petrella · Scala Days Berlin 2016
• 32mMachine Learning with Apache SparkTomasz Lelek · JDD 2016
• 44mApache Spark and ML workflowsJuliet Hougland · Jazoon Zurich Autumn 2017
• 50mDeep Learning and NLP with SparkAndy Petrella · Scala Days Berlin 2016
• 55mGetting Started with Spark and MLRob Richardson · Developer Summit 2020
• 24mMaking Spark ML Models Portable Know Your OptionsMatthew Tovbin · Scale By The Bay 2018
• 26mMLbase: A Distributed Machine Learning SystemTim Kraska · Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013)