Using Dask to Distribute Machine Learning Jobs
summary
Claims daily model training and backtests speed up by about 50% faster by distributing per-region training with Dask while keeping data small enough for a laptop. Uses Dask + pandas + scikit-learn, runs on EMR/YARN with EMR/YARN with zipped virtualenvs to distribute environments, supports threads or processes, worker memory watchdogs and a web UI, and includes light wrappers to fall back to serial runs. Achieves local development parity with cluster runs and successfully scales from laptop to tens or hundreds of nodes.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:22Order Volume Forecasting
- 3:48Design Goals
- 14:03Light Wrappers
- 14:31Deep Integration with Scikit-Learn
- 15:49The Takeaways
