Deploying Dask
summary
Deploying Dask is messy but solvable: an Optuna + XGBoost demo requires a MySQL replacement, spins up 20 machines on AWS, and surfaces a profile where 97% overhead hides the 3% training work. Real fixes use concrete tools: dask-gateway, dask-kubernetes, helm charts, dask-cloudprovider, or managed options like Coiled, Saturn, QHub, EMR/Dataproc. Key points are reproducible software shipping, secure network/auth, logs/telemetry for debugging, and spot vs on‑demand cost policies that actually reduce bills.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:49Dask deployment challenges
- 9:16Borrowing machines and clusters
- 13:06Software installation across systems
- 16:38Network access and security
- 19:04Observability and debugging
- 23:58Cost management and optimization
- 25:41Dask deployment solutions: Nabari, Saturn, Kubeflow
- 29:32Coiled platform for Dask
- 35:09Summary
- 36:13Q&A
related talks
28mDeploying Dask
26mTerraforming Jupyter and Dask: How to Get Your Own Cloud Data Science Platform on the Cheap
39mPyData in Production: Lesson Learned from Various PyData Deployment Strategies
44mUsing Dask for Parallel Computing in Python
29mBuilding Large Scale ETL Pipelines with Dask
57mCloud Adoption Patterns
38mData Secrets from a Platform Engineer
38mDeploying Database Clusters in the Cloud