Machine Learning at Scale
summary
scikit-learn falls short on very large datasets so the stack uses PySpark and MLlib on AWS with HDFS (optionally managed via Flint Rock) to run distributed training. example code builds a random forest on 26 classes from 16 numeric features with a ~70/30 train/test split, trains on a cluster, predicts and prints accuracy. setup details include installing a C compiler and numpy on nodes and the practical data flow of copy from S3 to HDFS before running Spark jobs.
AI-written summary · about
chapters (6)
- 0:00ML at scale intro
- 3:58Spark & RDDs crash course
- 10:09MLlib overview & training
- 13:05Classification pipeline
- 18:51Deploy models to AWS
- 23:17Q&A on scaling
related talks
50mApache Spark vs Machine Learning
17mMachine Learning at the Speed of Light with Spark
27mMachine Learning on big data in security applications
31mFrom Pandas to Spark
23mRethinking scalable machine learning with Spark ecosystem
44mApache Spark and ML workflows
35mPySpark : Combining Machine Learning & Big Data
31mProductizing Machine Learning over Big Data with AWS tools