Canary Deploys with Istio: Lessons Learned
summary
Built an automated canary pipeline that uses Istio + Spinnaker + Kayenta on Kubernetes with metrics in DataDog, deploying a baseline and a canary and routing traffic at a 90/5/5 split to compare 95th‑percentile latency and error counts. It automatically promotes or rolls back a release; it caught a bad DB query in production before wide impact. A race in cleanup where routing pointed 5% at destroyed pods causes a real outage, so pipelines now keep canaries alive and run a two‑stage (10min then 50min) analysis.
AI-written summary · about
chapters (24)
- 0:00Introduction and agenda
- 1:54Problem with current releases
- 3:17What are Canary deployments
- 4:45Why Canaries scale better
- 6:14How Canaries change deployments
- 7:17Goals for production pipelines
- 8:18Limiting noise in experiments
- 10:02How we deploy at PlanGrid
- 12:13First failed attempt
- 14:16Infrastructure changes enabling success
- 15:21Solving the three challenges
- 16:28Istio traffic management setup
- 17:12Metrics and Kayenta integration
- 19:01Configuring Spinnaker for metrics
- 19:44Creating canary configs
- 21:58Judgment and rollback logic
- 23:22Results in production
- 24:24Major incident and fixes
- 26:53Human challenges and false negatives
- 28:21Onboarding apps and debugging
- 29:45Tuning canary configurations
- 31:33Break glass deployments
- 32:16Next steps and future plans
- 35:05Q&A and closing
related talks
01h 04mThe Fine Art of Canary Deployments
30mVisualizing Canary Rollouts with Istio and Helm
51mCutting edge Continuous Delivery: Automated Canary Analysis through Spring based Spinnaker
34mEnabling Invisible Infrastructure Upgrades with Automated Canary Analysis
14mCanary Deployments for Infrastructure Upgrades
10mTest in Prod With Canary Everything
43mUsing Kubernetes, Spinnaker and Istio to Manage a Multi-cloud Environment
15mYahoo Case Study: Automated Canary Analysis using Spinnaker Kayenta