Pods Everywhere! InterLink: A Virtual Kubelet Abstraction Streamlining HPC Resource Exploitation
Talkat Cloud Native AI Day Europe 2024 · on Tuesday, 2024-03-19· at 11:05· Pavilion 7 | Level 7.3 | Room S03·
summary
An open source extension of Virtual-Kubelet called interLink exposes a Kubernetes API that maps Pod requests to remote runtimes via an Interlink API server and provider plugins, tested successfully against EuroHPC Vega and Jülich sites with a live deployment using 90 GPUs at Vega. It supports SLURM/Apptainer or local container runtimes, an installer plus a Python SDK for writing plugins, MLflow and notebook use cases (notebook connectivity via SSH tunnel), and currently limits volumes to ConfigMap, Secret, and emptyDir.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:45INFN computing challenges
- 3:54InterLink goals
- 6:22Kubernetes API solution
- 9:13Virtual Kubelet concept
- 13:27InterLink architecture
- 17:05Full stack and plugins
- 18:54Euro HPC centers Vega Julich
- 20:19Deployment and virtual node
- 21:45MLFlow pipeline OSCAR
- 24:16Jupyter integration
- 25:01Future development
- 26:48Conclusion
- 28:57Q&A
related talks
05mHow To Run Kubernetes Pods On My Slurm-Based HPC Center
12mLightning Talk: {”spec”: “nodeSelector”: "EuroHPC"}
30mThousands of Virtual Kubelets: 1-to-1 Mapping a Supercomputer To Kubernetes With Supernetes
30mBuilding a Global Supercomputer with Virtual Kubelet
36mIntro: Virtual Kubelet
35mDeep Dive: Virtual Kubelet
36mIntro: Virtual Kubelet
39mSlinky: Slurm in Kubernetes, Performant AI and HPC Workload Management in Kubernetes