Reproducible software provisioning for high performance computing & research software engineering
summary
Institute runs Spack package manager to reproducibly build and expose a full HPC stack via environment modules, with 600 packages across six architectures, tool chains combining compilers and MPI and about 30–40 packages per toolchain, automated from git-tracked specs into modules, local source and binary mirrors and a small Python installer; it works but faces concretization conflicts, flaky upstream mirrors (Dropbox, Intel), occasional package bugs and some OS header dependencies.
AI-written summary · about
chapters (17)
- 0:00Reproducible software provisioning
- 1:29Traditional installation challenges
- 3:35Introducing spec package manager
- 5:00Live demo: Zlib with spec
- 7:08Managing package versions
- 9:57Multiple versions with hashes
- 13:11Using packages and cowsay
- 13:33Advanced spec features for HPC
- 15:38Spec for Python dependencies
- 16:01Contributing packages to spec
- 17:28Why spec fits HPC
- 18:53Our HPC setup with spec
- 20:17Compiling for multiple architectures
- 21:20Dependency resolution challenges
- 22:49Demo: cowsay after install
- 24:18Summary and resources
- 25:04Q&A
related talks
07mBuild all the things with Spack: a package manager for more than C++
31mScience Reliability Engineering for High Performance Computing
48mIntegrating upstream projects downstream
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
26mOpenHPC on Ansible
27mRelease Management with Devpi
32mSoftware Supply Chain with the Yocto Project
49mDeveloping Research Infrastructures the DevOps way