Building Consul at Scale for Brownfield Infrastructure
summary
CrowdStrike replaces east‑west load balancers by deploying Consul across a hybrid fleet of 95,000 nodes, split into 40 Consul clusters and seven Vault clusters, partitioning gossip per AZ so clients bind to a local cluster and using prepared queries for localized failover. Certificates are issued via Vault PKI and Terraform with Vault agents rotating certs, Chef bootstraps nodes, and Terraform governs policies and tokens. Monitoring (Prometheus/Grafana) and automated upgrades prove the design viable, while Vault lease spikes and gossip connectivity are the main operational hazards.
AI-written summary · about
chapters (22)
- 0:00Introduction and background
- 2:25Use case for Consul at scale
- 5:15Network infrastructure challenges
- 7:25Provisioning with Terraform
- 8:33Sizing and performance considerations
- 9:57Security framework and Vault integration
- 14:13Access control and ACLs
- 15:41Gossip encryption and secrets
- 16:24Post-provisioning Vault configuration
- 17:47Chef cookbooks and testing
- 18:51Fetching initial secrets
- 20:36Connecting clients to clusters
- 22:00Onboarding and self-service model
- 24:47Automating upgrades and restarts
- 26:31Packaging binaries and supply chain
- 27:55Monitoring dashboards and metrics
- 32:47Log aggregation and alerts
- 35:33Learning normal behavior
- 36:54Deployment and rollout strategy
- 37:37Common problems and solutions
- 43:54Backup and recovery practices
- 44:36Future plans and automation
related talks
28mObserving the HashiCorp Ecosystem From Prometheus
42mScaling Resilient Systems: A Journey into Slack's Database Service
28mBootstrapping a Multi-DC Cloud Native Observability Stack
30mFrom hand waves to real data: Measuring the business impact of implementing security automation
13mAutomate an Observable Runtime with HashiCorp and Ansible
27mNomad Best Practices for Reliable Deploys
53mImplementing Kubernetes on with confidence
41mScaling Reliability Engineering with Tools