Scaling Scikit Learn
summary
A scikit-learn-based backend runs training on EC2 (training data must fit memory) and pushes predictions over Redshift tables by splitting Redshift exports into 6.2 GB CSV shards and scoring them in parallel on many EC2 instances, achieving roughly constant wall-clock time given enough nodes. The system uses S3 for staging, Docker containers for reproducible environments, a custom joblib backend for remote parallelism, chunked reads with retries, and Docker + semantic versioning to preserve reproducibility.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:56Scikit-learn and its API
- 10:22Cloud infrastructure needs
- 14:42Large data with Redshift & S3
- 21:02Parallel processing with joblib
- 27:19Cloud training & prediction
- 31:58Q&A
related talks
35mAccelerate deep learning in the cloud with custom ML environments
32mMaking Happy Modelers: Build and Maintain...
27mMachine Learning at Scale
18mUsing Dask to Distribute Machine Learning Jobs
28mScaling Machine Learning Training and Inference
31mProductizing Machine Learning over Big Data with AWS tools
48mData warehousing reinvented for today’s needs
39mImplementing distributed grid search for deep learning using scikit learn and joblib