HPC Multi-Cluster Database Deployment: Service Level and Availability Driven Refactoring
summary
Los Alamos runs a multi-cluster Slurm-backed scheduler using 20,000-node Trinity and other clusters (30–2,500 nodes) with MariaDB Enterprise for mission-critical decisions in seconds. A primary‑replica, high‑RAM, SSD‑tuned database with TCP tuning and staged pre‑prod virtual clusters handles outages: spools survive ~1.5 days of DB downtime, eight scheduler outages (2017–2022) force fixes—four were storage shortfalls fixed by sizing up storage and automation. Backups use MariaDB Backup in ~35 seconds and experiments include MaxScale for added resilience.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:10About LANL and Cray Supercomputing
- 5:12Terminology
- 6:36Scheduler Service Definition
- 9:39Timeline
- 11:01Uptime track record 2017-2022
- 17:28Mitigations
- 18:20Multicluster Slurm DB project
- 20:37Learnings
- 29:47Q&A
related talks
41mHTAP with MariaDB Platform
58mDatabase High Availability
25mThe future of databases: Combining operational and analytical data at massive scale
51mMariaDB Scaling - From Single Instance to Multiple Clusters
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
34mSailing the Database Seas: Applying SRE Principles at Scale
58m600k QPS on MySQL Galera cluster
16mPower Hour, Managing Databases at Scale – Bala Kuchibhotla