Scalable Data Science in Python and R on Apache Spark
summary
Concrete claims and tools: PySpark and SparkR expose the JVM-backed execution so most heavy lifting runs in the JVM and avoids language penalty; Spark ML Pipeline provides transformer/estimator stages, cross validation and model export; data-transfer pain from row-by-row UDFs is mitigated by Apache Arrow and vectorized UDF efforts; deep learning examples ship working integrations like BigDL with GPUs, CaffeOnSpark and TensorFlow-on-Spark, mapPartitions-based data pipelines, MNIST CNN training and group-wise gapply workflows for scalable hyperparameter search.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 1:04Spark overview and cluster architecture
- 3:53Spark use cases and workflows
- 4:57Python API for Spark
- 6:25R API for Spark
- 7:26Language bindings performance
- 8:11Spark in ML pipelines
- 12:49ML pipeline concept and stages
- 15:42Model parallelism and scheduling
- 16:46User Defined Functions in PySpark
- 19:39SKLearn library for scikit-learn
- 20:44CaffeOnSpark for deep learning
- 21:49BigDL for distributed deep learning
- 22:55TensorFrames for TensorFlow
- 23:37MLlib model enhancements
- 24:43Deep Learning library overview
- 26:34SparkR pipeline model
- 28:01SparkR formula syntax
- 30:09SparkR UDF support
- 31:34SparkR random forest example
- 32:37SparkR performance tips
- 34:06SparkR Lapply
- 35:10TensorFlow R binding
- 36:37SparkR alternatives
- 37:40Partitioning strategies
- 38:24Q&A
related talks
44mData Science at Scale with Apache Spark
11mScaling R with Spark
22mBringing Deep Learning into the Big Data Analytics
39mAccelerating Data Science with RAPIDS
16mPetabytes of data: How to use R at scale
36mScaling data workloads using the best of both worlds: pandas and Spark
23mRethinking scalable machine learning with Spark ecosystem
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons