Building a Scalable Data Science & Machine Learning cloud using Kubernetes
summary
Built a Kubernetes POC that runs RStudio and Jupyter in containers, uses Docker images stored in GCR private registry, authenticates via Active Directory, and exposes notebooks with an wild-card DNS + ingress. Architecture uses multi-AZ clusters with odd-numbered etcd replicas, dedicated control/ingress/storage nodes, EBS/EFS tradeoffs noted, and resource limits with fragmentation issues. The custom GUI layers on Rancher to let data scientists self‑service; the POC works but surfaces storage and scaling edge cases.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:28Meeting the team
- 1:00The current situation
- 1:51New platform requirements
- 5:28Kubernetes
- 9:50Demo
- 13:37Feedback
- 17:04Kubernetes Rancher Demo
- 19:22Why Im here
- 20:24Cluster Architecture
- 27:23Container Registry
- 28:43Studio and Jupiter
- 29:20Multiple Availability Zones
- 30:42Wildcard DNS
- 32:57Resource Limits
- 33:58Summary
- 35:10Contact
- 35:22Thank you
related talks
02h 42mBuilding an Open Source Cloud for Data Scientists from A to Z
31mAn Adventure in Multi Cloud with Kubernetes and Distributed Data
14mBuilding a high-performance analytics ecosystem on Amazon EKS
13mDesigning an Autonomous Workbench for Data Science on AWS
36mMulti-Cloud Machine Learning Data and Workflow with Kubernetes
41mBuilding & Operating a Large-scale HPC AI Cluster on Kubernetes
47mCloud Native and Sustainable, Reproducible Scientific Computing
39mHow We Hosted 200+ Data Scientists in a K8S Cluster