Science Reliability Engineering for High Performance Computing
summary
Claims an SRE-driven stack called OpenCHAMI replaces hands‑on vendor pets with a containerized, API‑driven boot and lifecycle system built in Go, using Podman, builda-produced squashfs images, GitLab/GitHub CI and sigstore signing. Targets are explicit: exact reproducibility, parallel bandwidth and lowered meantime‑to‑recovery. At LANL scale—~6,000 nodes, ~40–50 management nodes, 600,000 MPI ranks and a ~50 PB file system—installs drop from weeks to a day, CI runs on commit, and single‑node failures become less catastrophic.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:58High performance computing
- 4:27HPC challenges and benefits
- 8:45HPC reliability challenges
- 15:22SRE for HPC systems
- 19:32OpenChami stack
- 24:05Redesign lessons learned
- 26:50New approach benefits
- 28:34Future plans and goals
- 31:02Takeaways and closing
related talks
26mReproducible software provisioning for high performance computing & research software engineering
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
41mScaling Reliability Engineering with Tools
25mBuilding a highly scalable system to assure Business Resilience
13mProviding Continuous Integration at CERN
46mChaos Experiments - Datacenter Stress Testing
15mHow Our SREs Safeguard Nanosecond Performance—at Scale—in an Environment Built to Fail
36mShifting Performance Engineering Left