Canary Deployments for Infrastructure Upgrades
summary
Frequent infrastructure upgrades broke monitoring and secrets (Prometheus twice, Kubernetes/Vault once), so the response is an automated canary/blue‑green hybrid: spin up a new Prometheus statefulset (Prometheus Operator or RW PVC), deploy 100% to new statefulset, run existing Prometheus queries to ensure metrics return sane values, then trigger app CI/CD pipelines to validate dependent services (Spinnaker, Linkerd, Jenkins). Failures (renamed metrics, lost Helm annotations) surface immediately and cause rollbacks. Approach works for Prometheus and OpenCost; Kubernetes upgrades use a new cluster with the same validations before cutover.
AI-written summary · about
chapters (11)
- 0:00Infrastructure upgrade failures
- 1:04Detecting issues in production
- 2:07Testing complex customer configs
- 3:11Canary and blue-green strategies
- 4:14Hybrid canary deployment plan
- 5:39Validating with app deployments
- 6:46Prometheus upgrade experiences
- 7:50Unexpected benefits of approach
- 9:57Validating Kubernetes upgrades
- 11:27Redeploying apps safety
- 12:51Handling stateful apps
related talks
01h 04mThe Fine Art of Canary Deployments
04mCanary Deployment With Prometheus
43mCanary Deploys with Istio: Lessons Learned
34mEnabling Invisible Infrastructure Upgrades with Automated Canary Analysis
30mVisualizing Canary Rollouts with Istio and Helm
35mReduce Risks With Continuous...
42mMetrics Driven Blue-green Deployments using Spinnaker’s Cloud...
46mProgressive Deployments with Consul Service Mesh