How We Hosted 200+ Data Scientists in a K8S Cluster
summary
Describes deploying a scalable JupyterHub environment on Kubernetes to host over 200 data scientists, focusing on easy scaling, moderate fault tolerance, and centralized management. Covers the technologies Kubernetes and JupyterHub and how they were configured to provide a convenient multiuser workspace for data science teams.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:19Early solution problems
- 8:20JupyterHub on Kubernetes
- 14:03Team personal environments
- 17:15Resource isolation by namespaces
- 20:01Spark networking issues
- 22:57MetalLB load balancer problems
- 27:10User data migration
- 28:37Monitoring and metrics
- 31:10Summary
related talks
27mDeploying JupyterHub with Kubernetes
44mEmpowering Data Science with Kubernetes | Samuele Chiocca and Andrea Rota
05mHow to Simplify Data Scientist’s Day in Huge Company Using Kubernetes
37mJupyterHub from the Ground Up with Kubernetes
32mDemocratizing Distributed Computing with Dask and JupyterHub
26mTerraforming Jupyter and Dask: How to Get Your Own Cloud Data Science Platform on the Cheap
35mLarge Scale Teaching Infrastructure with Kubernetes
12m⚡ Lightning Talk: Data Science Environments in Seconds: Scaling Jupyter Notebooks in Kubernetes