Robert Ross (Fire Hydrant) - Spinnaker Summit 2019
summary
A Ruby on Rails 4.2 bug creates unbounded prepared statements in Postgres causing DB memory growth and swapping; the quick fix is a bash loop that deletes random pods every 15s (the ad hoc Killah for Thrilla) which keeps the site up but becomes an invisible single point of failure. The durable fix uses Spinnaker cron triggers and a pod annotation patch to force rolling restarts, giving auditability, RBAC and parameterized levers and knobs via pipelines and webhooks while warning that automation adds its own complexity.
AI-written summary · about
chapters (6)
related talks
24mGrowing Pains: Taking off the training wheels at Salesforce
41mAfterlife Tales
05mIgnite – Adrian Todorov
39m30 minutes of Failure
19mSpinnaker @ Upside: Getting Started, Establishing Culture & Deploying Microservices with Confidence
43mStack Smashing
27mIt's Down! Simulating Incidents in Production
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)