Lightning Talk: How to Avoid a Kubernetes Doom Loop
Lightningat ArgoCon Europe 2023 · on Tuesday, 2023-04-18· at 11:55· Elicium Building | Elicium Ballroom 2 [Argo Workflows]·
summary
A misconfigured pod causes automated deletes to loop, flooding a GCP cluster running 16K Argo Workflows across >165 nodes and creating 18,000 workflows and deployments stuck pending, exhausting instances and preventing any app create or delete. Root cause is a Python variable typo in an Ingress (singular vs plural), cleanup required custom scripts and 6.5 hours cleanup. Remedies implemented: separate system node pool, Argo Events rate limits and parallelism, semaphores, retry limits, PodDisruptionBudgets, priority classes, metrics and workflow-default config maps.
AI-written summary · about
chapters (4)
related talks
12mLightning Talk: Using Argo Workflows to Curate Chaos Engineering wi...
21mHow to Not Destroy Your Production Kubernetes Clusters
38mArgo and Tekton: Pushing the Boundaries of the Possible on Kubernetes
09mLightning Talk: Optimizing Latency in Argo Workflow Submissions
34mReading 100+ Kubernetes Post-Mortems - lessons learned
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
47mPostmortem: A Modern Approach to Managing Kubernetes Clusters
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster