An almost perfect match: cloud native and scientific computing
summary
CERN migrates decade‑old analysis into containers and Kubernetes, runs a 70 TB, 25,000‑file dataset as 25,000 single‑core jobs on a 25,000‑core cluster to finish in about five minutes and cost ~$250, reproducing Higgs plots; machine‑learning job scales linearly from 1 to 128 GPUs via Kubeflow and yields ~100× faster training; challenges are image distribution (use of sregz/remote snapshotter), multi‑cluster queuing and co‑scheduling (volcano/armada), and data‑aware scheduling for petabyte gravity rather than lack of feasibility.
AI-written summary · about
chapters (9)
- 0:00Introduction to cloud native and science
- 2:54Why containers matter for science
- 8:55Analysis preservation with containers
- 13:49Scaling machine learning workloads
- 18:42Consolidating IT and scientific workloads
- 23:40Efficiency in software distribution
- 25:26Advanced scheduling needs
- 29:19Summary and future of cloud native science
- 30:22Q&A session
related talks
19mKeynote: Building Bridges: Cloud Native and High Performance Computing
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models
35mState of Containers and the Convergence of Hyperscale, BigData and HPC
29mStreamlining Competitive Data Science at CERN: Running ML...
46mMachine Learning with Kubernetes
01h 21mHPC in the cloud
20mKeynote: Reperforming a Nobel Prize Discovery on Kubernetes
20mHow the cloud changed the way I do analytics and research