Machine Learning with Apache Spark
summary
Shows how to take forum text and build ML models in Apache Spark (using Spark MLlib and the DataFrame API) to infer which user likely authored a post. Covers NLP and Bag-of-Words feature extraction, preparing input data for modeling, and training Gaussian Mixture Models and Logistic Regression to produce probability-based authorship predictions.
AI-written summary · about
chapters (33)
- 0:00Intro
- 2:09Tokenization
- 2:53Remove Stop Words
- 3:31Lemmatization - Morphological Analysis
- 4:33Load forum data
- 5:17Tokenize and Stop Words
- 6:57Transforming text into vector of numbers
- 7:41Bag-of-Words
- 9:51Skip-Gram
- 10:56Spark Word2 Vect
- 12:33Machine Learning
- 13:46Used techniques
- 13:57I. Logistic Regression
- 14:24Logistic Regression example input
- 14:58Chart
- 15:13Example result
- 15:35II. Gaussian Mixture Model
- 16:14Graphic representation
- 16:53Next steps to build model
- 17:10Input data for our algorithms
- 17:41Logistic Regression model per author
- 19:06Area under ROC
- 19:23Interpreting measures
- 19:49Prepare labeled data
- 22:04Model validation
- 23:15Time of day distribution for author X
- 24:26Preparing data for GMM
- 25:01Creating GMM
- 25:31Evaluating Logistic Regression with GMM model.
- 26:22Find author for post
- 26:36Test run
- 28:58How it could be used?
- 29:41Thank you, Questions?
