Data Science at Scale with Apache Spark
summary
Argues that Apache Spark replaces MapReduce for Hadoop workloads and shows working code: a Scala inverted index in ~30 minutes that reads text files, flatMaps words, reduces by (word,path) to counts and sorts; demonstrates Spark SQL/DataFrame with the Catalyst optimizer and Parquet I/O; explains micro‑batch DStream streaming k‑means with 5 clusters and decay=1 for live training and prediction; mentions MLlib and GraphX for iterative ML and PageRank, and claims much better developer productivity and performance.
AI-written summary · about
chapters (13)
- 0:00The state of big data
- 2:20MapReduce and Hadoop
- 5:33Early Hadoop abstractions
- 9:11Scalding and functional data
- 11:52The rise of Apache Spark
- 13:59Spark advantages and features
- 17:37Spark streaming and mini-batch
- 20:28Implementing inverted index
- 27:44SQL support in Spark
- 30:39Spark DataFrames API
- 33:53Machine learning with Spark
- 36:40Streaming machine learning
- 40:01Graph processing with GraphX
related talks
45mFrom Big Data to Fast Data: Apache Spark
17mMachine Learning at the Speed of Light with Spark
11mHow Companies are Using Spark, and Where the Edge in Big Data Will Be
42mScalable Data Science in Python and R on Apache Spark
50mApache Spark vs Machine Learning
47mScala: The Unpredicted Lingua Franca for Data Science
01h 25mApache Spark for Big Data Processing
42mScala: The Unpredicted Lingua Franca for Data Science