Confidence with Chaos for your Kubernetes Observability
summary
A demo app intentionally leaks a 1 MB receive buffer on DNS resolver errors and a scheduled chaos mesh experiment injects DNS errors and CPU/memory stress, which drives container memory from ~200–300 KB past a defined SLO alert at 10 MB and makes OpenTelemetry traces grow about 5× (from ~2 ms to ~9 ms). Monitoring uses Prometheus with the Prometheus operator, Alertmanager, Grafana and Jaeger; alerts and dashboards show the failures and grouping issues.
AI-written summary · about
chapters (22)
- 0:00Introduction to talk
- 1:10Kubernetes monitoring challenges
- 2:32Prometheus for metrics
- 4:19Prometheus Operator benefits
- 5:43Service level objectives and alerts
- 7:51Alert management best practices
- 9:13Manual chaos testing
- 10:19Chaos engineering frameworks
- 12:47DNS failure memory leak case
- 15:13Live chaos demo setup
- 16:17DNS chaos experiment
- 18:59Automated chaos scheduling
- 19:20Alerts from chaos experiments
- 20:02Observability beyond metrics
- 22:32Logs and tracing importance
- 24:36OpenTelemetry overview
- 26:24Tracing with chaos experiments
- 28:10Future of observability
- 29:15Security and chaos engineering
- 30:00Chaos engineering limits
- 30:43Continuous observability workflows
- 31:50Q&A session
related talks
36mConfidence with Chaos for Your Kubernetes Observability
30mBring Your Chaos to Work Day
32mTesting Kubernetes apps' observability end-to-end
42mDistributed Monitoring: How to understand the Chaos
35mFrom Monitoring to Observability: Left Shift your SLOs with Chaos
39mMake Cloud Native Chaos Engineering Easier - Deep Dive into Chaos Mesh
45mSchrödinger's Pokémon: Observability for Chaotic Load Testing- 42mDistributed Monitoring: How to understand the Chaos