Building an Open Source Cloud for Data Scientists from A to Z
summary
An end-to-end open-source datalab is built and deployed on Kubernetes (kube-spray on OVH, 3 nodes with 8GB/2vCPU) and run with Helm charts, ingress-nginx, and cert-manager using Let’s Encrypt (DNS/TXT or ACME). The stack provides Onyxia web UI for creating per-user namespaces, Keycloak OIDC authentication, Jupyter and VS Code instances, S3/minIO buckets, and git-backed workflows, with practical notes on websocket/reload ingress issues, certificates, storage, quotas, and autoscaling.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 10:58Kubernetes cluster setup
- 1:09:27Basic Jupyter deployment
- 1:22:44Onyxia presentation
- 1:31:37Onyxia demo
- 1:54:40Adding authentication
- 2:19:45Q&A
related talks
36mBuilding a Scalable Data Science & Machine Learning cloud using Kubernetes
32mWhat if we built an open source cloud for data science?
43mHow to build your own cloud-native platform on IaaS clouds in 2021
39mHow We Hosted 200+ Data Scientists in a K8S Cluster
01h 02mDesigning an open-source observability stack for Kubernetes
22mYour Cloud. Your Rules. Building a Sovereign Cloud with Open Technologies
44mEmpowering Data Science with Kubernetes | Samuele Chiocca and Andrea Rota
31mAn Adventure in Multi Cloud with Kubernetes and Distributed Data