Deploying Dask
summary
Requesting 20 machines with two cores each on AWS provisions T3 mediums in 20–30 seconds, pip/conda packages and local Git code get installed, and the Optuna + XGBoost workload runs but profiling shows most time spent in coordination not training. Deploy questions are concrete: how software is shipped (Docker, shared FS, or packagesync), how TLS and gateways are configured (dask-gateway, dask-kubernetes), what telemetry is collected, and how spot/on‑demand mix and autoscaling control costs.
AI-written summary · about
chapters (10)
- 0:00Deploying Dask introduction
- 1:28Why Dask deployments are hard
- 7:47Distributed deployment challenges
- 8:09Borrowing machines for clusters
- 10:34Installing software across machines
- 14:28Securing cluster access
- 16:54Observability in Dask deployments
- 20:05Managing users and costs
- 21:52Scaling Dask deployments
- 24:42Coiled simplifies Dask
related talks
41mDeploying Dask
26mTerraforming Jupyter and Dask: How to Get Your Own Cloud Data Science Platform on the Cheap
44mUsing Dask for Parallel Computing in Python
16mReducing Operational Barriers to Model Training
47mDeploying Like a Boss: From 2 Servers to 20,000
01h 15mEasily deploy+maintain an open source data science platform on the cloud of your choice
42mThe Challenges Managing a Kubernetes-Based Machine Learning Infrastructure
46mDeploying and maintaining software for today's infrastructure