Pod Anomaly Detection and Eviction using Prometheus Metrics - David Benque & Cedric Lamoriniere
summary
A Kubernetes controller called CubeVisor uses Prometheus queries (you supply a PromQL returning 1 for healthy, other values for deviation) to orchestrate service‑level circuit breaking across pods; it implements a breaker and an activator with strategies like continuous value deviation, discrete/whitelist, retry+kill and periodic modes. Demo runs against a four‑pod flight‑price service, marks bad pods with label traffic=yes/no to remove them from service discovery, retries every 15s then kills after one retry, annotates pods with attempt counts, and successfully ejects and recreates unhealthy pods; code is open source.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 2:35Example
- 3:44Kubernetes
- 4:36Probes
- 7:54Circuit Breaker
- 9:14Retry Policy
- 12:53Custom Resource
- 16:50Service Selector
- 18:53Primitives
- 19:18Demo
- 19:59Dashboard
- 20:50Configuration
- 21:41Continuous Value Deviation
- 23:05Cuba Reserve Service
- 24:18Pod CTL
- 27:05Testing
- 30:07Annotation
- 31:13Controller
- 33:20Key takeaway
- 34:27Question
related talks
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...
37mProactive Kubernetes Security: Anomaly Detection and Runtime Alertin...
42mMaking Resilient, Self-Adaptive, And Autonomous Kubernetes Clusters
37mMonitoring Kubernetes with Prometheus
33mBringing Pod Security Admission to Your Kubernetes Distribution
32mTesting Kubernetes apps' observability end-to-end
29mProduction Cluster Monitoring and Remediation for High Reliability
35mAnomaly Detection for Cloud Native Storage