Best Practices for Kubernetes Runners
summary
F5 Networks runs GitLab CI for ~1,800 users and ~350–400k CI jobs/month and suffers an incident where a miswritten automation schedules pipelines exponentially — about 1,024 jobs after 10 iterations and up to 16.7 million jobs in 24 hours if unchecked. Remediation is to disable CI on the offending project, cancel pending jobs via GitLab Rails, and add monitoring against a read‑only Postgres queue with embedded playbooks. Enforce Kubernetes requests/limits (example concurrency of 200, build containers 2 CPU/6 GB) and add pod labels for debugging and metrics.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:20GitLab runners we manage
- 6:59What do you do?
- 8:35What did we learn?
- 11:24Kubernetes Limits
- 13:24Adding limits after the fact
- 18:23What happens when a user hits a limit?
- 20:29Labelling your workload
- 21:15Label limitations
- 23:03Summary
related talks
35mGitLab CI and Kubernetes, Build Test and Deploy Your Projects
35mGitlab Runners at Scale as a Service
55mCI/CD Best Practices with Kubernetes and GitLab
46mFrom 1,000 to 5,000 Users: Scaling GitLab Without (Too Much) Pain
36mBuilding a CI/CD driven infrastructure for managing Kubernetes clusters
13mProviding Continuous Integration at CERN
26mOptimizing Your CI Pipelines
22mEmbedded Software Development Efficiency Improvement with Docker and Kubernetes