Testing Kubernetes apps' observability end-to-end
summary
Shows how to spin a disposable local cluster with kind, deploy Prometheus and Alertmanager, and run unit + end-to-end tests that verify a Prometheus client metric increments (expect initial +1) and that recording rules and alerts fire when a deployment is scaled to zero. Uses a Prometheus linter to catch naming/label mistakes (many issues found), exercises recording rules for operator pod counts, validates handbook URLs, and demos alert state changes (pending/firing). Also suggests Chaos Mesh for fault injection.
AI-written summary · about
chapters (10)
- 0:00Kubernetes observability testing intro
- 1:03Disposable test environment setup
- 3:31Validating observability metrics
- 5:16Testing actionable alerts
- 11:11Demo setup and metrics validation
- 17:01Unit testing metrics rules
- 20:56End-to-end observability testing
- 26:19Chaos engineering testing
- 27:23Key takeaways and best practices
- 29:34Q&A on metrics and alerting
related talks
24mPractical Kubernetes Monitoring with Prometheus
36mConfidence with Chaos for Your Kubernetes Observability
28mImplementing Observability in Kubernetes
37mMonitoring Kubernetes with Prometheus
37mKubernetes, Monitoring with Prometheus and Grafana
02h 43mPrometheus from A to Z
26mEnd-to-end monitoring with the Prometheus Operator
32mUnderstanding Observability in Kubernetes