Deep Learning and NLP with Spark
summary
Implements a Spark‑ready LSTM used for sentiment on IMDB reviews, training a two‑layer LSTM with 200 cells fed by 300‑dim word vectors and batched at 50, producing two output classes; uses a prebuilt word2vec Google News (1.5GB, expands to 4GB) model broadcast to executors and the DL4J + ND4J with C++ backend via JavaCPP to exploit CPUs/GPUs. Notebook demo wires Spark, streaming and visualizations; code is runnable but the live demo is offline — design and scaling work as described.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 1:20Machine Learning
- 2:29What is Deep Learning
- 5:35About Scary Mind
- 6:56Why is it not easy
- 8:21What our stack looks like
- 8:49Back Propagation
- 9:29Data Parallelization
- 10:50NLP
- 11:58Recurrent Neural Net
- 16:28Examples
- 18:48Word Tyk
- 22:13Autogenerated Shakespeare
- 23:19Sentiment Classification
- 23:42Thank you
- 23:58Spark Notebook
- 28:48Cluster
- 31:32Imports
- 32:41Google News
- 33:48DL4J
- 37:45Logistic Regression
- 38:40Creating the model
- 40:14Importing the model
- 40:40Data structure
- 41:38Review numbers
- 43:24Data
- 44:52Train
- 45:21Prediction
- 46:44The JVM
related talks
35mJVM based DeepLearing on IoT data with Apache Spark
01h 30mIntroduction to Deep Learning & Natural Language Processing
43mDetecting fake reviews at scale using Apache Spark and John Snow Labs
54mNatural Language Processing with Deep Learning and TensorFlow
40mScaling From Research to Production with Skymind DL4J and ScalNet
40mScala and Spark in the New Deep Learning Era
31mDeep Learning Techniques for Natural Language Processing
01h 59mDeep Learning and Modern NLP