Implementing distributed grid search for deep learning using scikit learn and joblib
summary
Grid search is implemented as a distributed pipeline that wraps a TensorFlow deep text classifier in the scikit-learn estimator API and parallelizes hyperparameter runs using a joblib custom backend on Amazon EC2. The claim is concrete: grid search wall time drops from about an hour on a laptop to roughly 15 minutes in the cloud. The code launches tasks to Docker containers, returns results via pending-result objects, and the distributed grid search demonstrably works.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:47Custom ML with TensorFlow
- 7:13TensorFlow basics
- 13:39Scikit-learn API
- 23:00Distributed grid search
- 28:44Evaluating search
- 34:06Key takeaways
- 35:40Q&A
related talks
35mScikit-Learn can do THAT?!
34mScaling Scikit Learn
18mUsing Dask to Distribute Machine Learning Jobs
35mAccelerate deep learning in the cloud with custom ML environments
44mApache Spark and ML workflows
37mDistributed Deep Learning on Apache Mesos with GPUs and Gang Scheduling
34mDemocratizing Machine Learning on Kubernetes [I]
01h 01mParallel Python Analyzing Large Data Sets