Downtime Budgets
summary
Adapts the error‑budget idea to HPC by tracking per‑cluster downtime budget (one 10‑hour DST per month → 30 hours per quarter ≈ 1.4% time) across ~20 clusters (≈36,000 nodes, 1.2M cores, 110PB). Prototype uses python scripts → MySQL stats DB → JSON → plotlist graphs to show burn rate, seasonal hits and a case that went to about −13 hours. Outcome: practical planning tool that exposes hidden costs (overnight outage burned a quarter), lets ops schedule facilities, lightning windows, dedicated user runs and technical‑debt work.
AI-written summary · about
chapters (26)
- 0:00Intro
- 1:30What is HPC
- 3:51The Lab
- 5:35Typical HPC Jobs
- 6:02HPC vs Web Engineering
- 6:49Metrics
- 7:38Utilization
- 8:39Optimization
- 8:52Maintenance
- 10:05Impact to Users
- 10:40Overview
- 12:20Goals
- 12:59Not Goals
- 13:26Initial Implementation
- 13:54Initial Implementation Graph
- 14:36Baseline Graph
- 16:24Power Outage
- 16:59What can we do
- 17:10Science tachometer
- 17:48Facilities work
- 18:35Lightning strikes
- 19:08Dedicated user requests
- 19:44Technical debt
- 20:28Challenge
- 23:10Community
- 24:01Measuring Metrics
related talks
51mWhat to Do When a Minute of Downtime Costs $100,000
19mLatency and Availability Error Budgets Done Right at Scale
26mImprove application resilience with chaos engineering
34m"The Upside of Downtime: How to Turn a Disaster Into an Opportunity"
30mReliable Observability at Scale: Error Budgets for 1,000+
39mSPACE and Beyond - Measuring Developer Productivity
26mUser Uptime in Practice
06mDevops Behavioral Economics: Displayed...