How we Orchestrate 7M+ Builds/Month with Kubernetes - Danielle Tomlinson
summary
Orchestration runs 7M+ builds/month by running Nomad servers inside Kubernetes while build agents run on AWS autoscaling groups, using Helm and Terraform to deploy per-cluster namespaces and statefulsets. Nomad handles very fast batch scheduling; Kubernetes provides rolling updates, readiness checks and operational stability. A custom garbage‑collector and client health agents fixed a severe GC-induced stall that once required nuking state; recent Nomad batching removes that pain and the setup is stable in production.
AI-written summary · about
chapters (29)
- 0:00Intro
- 2:28The industry was changing
- 3:30Requirements
- 3:45Build Infrastructure
- 5:56Operational Simplicity
- 6:21Service orchestration
- 7:42Fast Scheduling
- 8:00Pluggable
- 8:27Cooperative API
- 9:17One size doesn't fit all?
- 11:36Kubernetes Extensibility
- 11:56Nomad is part of our product
- 12:49What is nomad?
- 13:37Persistence
- 14:35Multiple clusters
- 15:26Scaling Clients
- 15:53Draining Clients
- 16:21Health Checks
- 16:38Multi-Cluster
- 16:59Terraform, terraform, terraform
- 17:19Helm
- 18:03Namespaces
- 18:18Service Discovery
- 19:26The Symptoms
- 20:25Declaring Bankruptcy
- 20:37Follow up
- 20:59The next day
- 21:17Analysis
- 23:12Nomad 0.8
related talks
30mA beginner's journey: Scaling self-hosted platform infrastructure with Nomad
12mNomad: Lightweight Orchestration That Complements Kubernetes
31mOrchestration with Nomad
44mNomad, orchestration made in HashiCorp
33mAbstracting Nomad: Patterns for Deploying to Nomad at Scale
35mLiving the Nomadic life
53mBootiful workload orchestration with Hashicorp Nomad
04mAdelbert Chang Interview