Unlimited Data Science Libraries, One Container Image, No Insta...
summary
Kubernetes hosts a read-only shared rocks volume that provides thousands of prebuilt scientific packages via environment modules so notebook and RStudio pods dynamically load specific versions instead of shipping many images or a single 20GB image. Packages are built with EasyBuild (≈2,500 packages supported) and generate module files; builds optimize for target hardware, installations persist outside the container, and files are fetched on demand. A compatibility layer and a European environment project aim to make the same module set usable across clusters, laptops, and HPC.
AI-written summary · about
chapters (6)
- 0:00Start
- 2:41Open Data Hub
- 8:49Increase in Compute Power
- 20:52Environment Modules
- 30:47Future Work
- 32:02File System Layer
related talks
34mThe Data Analytics Platform or How to Make Data Science in a Box Possible
28mProgress For Big Data In Kubernetes
47mCloud Native and Sustainable, Reproducible Scientific Computing
25mStreamlined Efficiency: Unshackling Kubernetes Image Volumes for Rapid AI Model...
24mDynamic Big Data in the Cloud with Kubernetes
35mSimplification of Data Pipelines by Containerization
30mWhen Things Get Big
14mBuilding a high-performance analytics ecosystem on Amazon EKS