Zero production incidents. How to use an unpopular solution on a project
summary
A running platform built on Nomad, Consul, and Vault achieved zero production incidents for 20 months by using AWS spot instances (about 3× cheaper), blue–green upgrades, and GitHub-backed Vault policies to rotate credentials every 12 hours. The stack removes auxiliary stateful helpers, accepts some availability tradeoffs to improve stability, and reports metrics such as deploy frequency, restart rates, and the near-elimination of infrastructure incidents.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:24Team and platform
- 3:34Spot instances and envs
- 4:18Nomad Consul Vault arch
- 6:06Nomad selection and benefits
- 7:34Consul Vault integration
- 9:19Dynamic secrets and security
- 11:05Deployment monitoring and alerts
- 13:11Dev tools and features
- 15:2020 months no incidents
- 17:56DevOps productivity
- 20:46Monitoring and logging
- 22:32Stability trade-offs
- 24:02Cluster resources and stability
- 25:45Control plane load
- 27:29Nomad vs Kubernetes
- 29:59Networking limits
- 31:00Dependency management
- 32:26Why Nomad fits tasks
- 34:12Community and evolution
- 35:58When not to use Nomad
- 37:01Q&A
- 40:09Blue-green deployments
- 41:34Consul and Nomad Q&A
related talks
40mStrategies for Zero Down Time, Frequent Deployments
36mPractical Cloud Native: What Works, What Doesn't
40mInfrastructure as Code in the Wild
31mHow We Went From Two Major Outages to 99.98% Reliability in 6 Months
27mVault At Scale
30mZero Down Time Operations: How to Get Atleast 3 9's of Uptime
56mGetting all the 99.99(9) you always wanted
44mServerless Architecture on Production: Good, Bad and Ugly Parts