Rethinking scalable machine learning with Spark ecosystem
summary
Argues that an end-to-end ML pipeline runs inside the Apache Spark ecosystem by doing ETL, feature engineering and model training at scale, then bridging into deep learning with Petastorm to convert Spark DataFrames into TensorFlow Dataset or PyTorch DataLoader. Shows a working demo that loads images as binary, writes uncompressed Parquet to avoid small file issues, converts to a distributed dataset and fine-tunes MobileNetV2 with distributed TensorFlow; code and configs for cache dir and converters are on the public GitHub.
AI-written summary · about
chapters (6)
related talks
39mMachine Learning with Scala on Spark
35mPySpark : Combining Machine Learning & Big Data
39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and cons
33mDeploying and Scaling Spark ML and Tensorflow AI Models
34mDeep Learning at Scale
17mMachine Learning at the Speed of Light with Spark
44mApache Spark and ML workflows
42mScalable Data Science in Python and R on Apache Spark