Streamlining Competitive Data Science at CERN: Running ML...
summary
CERN builds a private ML-challenge platform that runs participant code as Kubeflow pipelines components so data download, distributed training, scoring and leaderboard submission are all orchestrated on-prem. The stack uses Katib, KServe, Argo, Vault and OIDC integrations and heterogeneous GPUs (T4, V100, A100, H100) with per-user quotas of 1 GPU and 30 GB of storage. Submissions arrive as container or notebook submission images, scoring components are provided per-challenge, and the demo (Iris) completes end-to-end.
AI-written summary · about
chapters (10)
- 0:00CERN and physics goals
- 3:17ML in trigger systems
- 5:22Need for Kaggle-like platform
- 8:12CubeFlow infrastructure
- 11:07ML challenge workflow
- 15:42Challenge management UI
- 16:28Live demo of platform
- 22:43Data handling in pipelines
- 24:51Platform roadmap
- 27:25Q&A
related talks
24mA Better and More Efficient ML Experience for CERN Users
19mRunning ML Challenges on Kubernetes (Part 2)
29mA Scalable Platform for Training and Inference Using Kubeflow
31mBuilding and Managing a Centralized ML Platform with Kubeflow at...
37mA Day in the Life of a Data Scientist. Conquer ML Lifecycle on Kubernetes
31mKubeflow - Data Science on Steroids
36mAn almost perfect match: cloud native and scientific computing
51mMachine Learning using Kubeflow and Kubernetes