Deploying Python at Scale with Dask
summary
Dask claims to make NumPy/Pandas/Scikit-Learn scale to 100+TB datasets and runs on Kubernetes, Yarn and cloud VMs; deployments actually use containers, TLS certificates and credential forwarding, and integrate with Prometheus for logging and billing. Real problems are uniform Python environments, resource sharing, authentication and runaway cost from interactive bursts; practical fixes include adaptive scaling, enforced idle time and cost tracking, and the Dask Gateway for secure multiuser clusters, which operators report works in production.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:10Why this talk
- 4:11Data Science Libraries
- 7:46Task
- 8:21Desk
- 10:01Environment Management
- 10:13Uniform software environments
- 11:22Data science vs IT
- 12:07Resource sharing
- 13:24Access
- 14:25IT Professional
- 14:46Credentials
- 15:51Security
- 17:08Costs
- 17:54Avoid Track Optimize
- 20:00Cost
- 20:50Conclusion
- 21:31Friendly Resource Managers
- 22:21Managed Solutions
- 22:31opinionated solutions
- 22:48Coyle Computing
- 23:01Managed Services
- 24:02Summary
related talks
01h 17mScaling PyData with Dask
01h 22mUsing Dask for Parallel Computing in Python
38mScalable Machine Learning with Dask
01h 23mData Science at Scale with Dask
29mScaling Data Workloads with Cluster Managers
23mData Science at Scale Using dask and Numba
01h 03mDask: Next Steps in Parallel Python
01h 26mDask tutorial