PySpark : Combining Machine Learning & Big Data
summary
Shows how to fuse machine learning and big data by running PySpark pipelines and MLlib algorithms at scale, ingesting from sources like MongoDB and HDFS and deploying on YARN or EC2. Demonstrates the PySpark + MLlib workflow: transformers, estimators, pipelines, model persistence (save/load), and common algorithms (logistic/linear regression, k‑means, collaborative filtering). Explains the Py4J bridge and executor/driver architecture, cites RDD→DataFrame→Dataset evolution (2011/2013/2015) and the 90% recent‑data growth to justify scale.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:39About Me
- 2:46Pen Paper
- 3:06Big Data
- 4:55Machine Learning
- 6:52Why fuse
- 7:06Data Growth
- 8:49Fusion
- 9:16Apache Spark
- 9:44What is Apache Spark
- 12:14RDD Data Frame Data Set
- 13:56RDD Features
- 14:41Spark Architecture
- 17:50Big Data Processing
- 19:36Spark Context
- 22:30PySpark Architecture
- 27:39ML Library
- 32:06Pipeline
- 32:31Spark Machine Learning Library
- 34:15Conclusion
related talks
50mApache Spark vs Machine Learning
23mRethinking scalable machine learning with Spark ecosystem
17mMachine Learning at the Speed of Light with Spark
04mThe machine-learning renaissance
31mFrom Pandas to Spark
39mMachine Learning with Scala on Spark
44mData Science at Scale with Apache Spark
01h 25mApache Spark for Big Data Processing