Kubernetes the New Research Platform
summary
Kubernetes is deployed on Compute Canada OpenStack with Kubespray, Terraform and Ansible and used to run an Argo workflow that launches 30 seeded instances of the Salmon algorithm writing to a persistent volume. Compute Canada claims ~300,000 cores, 12 petaflops and 50+ PB; University of Michigan runs ~27,000 cores plus a 14,000‑core restricted cluster and three Kubernetes clusters (~10 nodes, ~512 GB RAM). Examples include Argo workflows, JupyterHub provisioning, MPI operator for distributed training.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:01$ whoami - Lindsey
- 2:52Why Kubernetes?
- 9:05Back to Salmon
- 10:26ATLAS Collaboration
- 12:25Compute Canada and CERN
- 13:50Salmon on Kubernetes
- 15:07Salmon runs
- 15:41Salmon Results
- 15:56Future of Kubernetes at CC
- 19:58ARC-TS Needs
- 20:59A few services..
- 22:55Why Kubeflow?
- 23:50The New Research Workflow
- 24:48Challenges
- 26:24Current User Distribution
- 27:41Future @ UofM
- 30:45Expected User Distribution
- 33:20Useful Links
related talks
19mKeynote: Building Bridges: Cloud Native and High Performance Computing
36mAn almost perfect match: cloud native and scientific computing
27mHarnessing Kubernetes for Scalable AI/ML Workloads
44mEmpowering Data Science with Kubernetes | Samuele Chiocca and Andrea Rota
41mBuilding & Operating a Large-scale HPC AI Cluster on Kubernetes
12mKubespray: Driving Cost-Efficiency for AI on Kubernetes
46mMachine Learning with Kubernetes
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models