Optimizing Workload Management in a Multi-Cluster Environment: A Case Study
summary
OpenShift CI runs over 1,000 GitHub repositories across about 10–12 build clusters, handling roughly half million tests per month by dispatching jobs periodically rather than in real time to preserve namespaces. A dispatcher takes inputs like cluster capabilities (ARM, GPU, internet access), collocation, execution history from Prometheus-based history, and cluster status to preassign jobs; ephemeral clusters support E2E tests and static assignments are being phased out while full infrastructure redundancy remains a gap.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:03Single to multi-cluster evolution
- 3:07Cluster strategies pros and cons
- 4:55Reasons for multi-cluster adoption
- 5:58Periodic workload dispatching
- 7:02Workload distribution inputs
- 8:46Dispatcher algorithm explained
- 10:33Cluster assignment and persistence
- 11:35Platform future evolution
related talks
38mRunning Multi-Arch Builds & Tests in OpenShift CI/CD
30mOptimizing costs across several OpenShift clusters
44mPerformance and Scalability Tuning Kubernetes for OpenShift and Docker
29mHow to Build a Cluster to Run Hundreds of Thousands of High-CPU/High-MEM Jobs
20mBenchmarking HPC workloads on OpenShift
42mCI/CD - Let The Robots Do Their Job!
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
49mCI for OpenShift: Prow, ci-operator and the future