Designing an open-source observability stack for Kubernetes
summary
Practical recipe for Kubernetes observability on AWS: use Prometheus + Grafana for metrics, ELK (Elasticsearch Logstash Kibana) for logs with Kafka or Kinesis as a buffer, and Jaeger tracing for distributed traces; persist telemetry in real time, run Elasticsearch with three masters and zone-aware shards, and keep the monitoring stack more available than production. Add Fluentd/Fluent Bit collection, threat enrichment from CloudTrail/GuardDuty/Suricata/Zeek for security, and cluster logs into patterns to cut noise; the integrated cloud service demo shows these pieces working end-to-end.
AI-written summary · about
chapters (21)
- 0:00Introduction and overview
- 1:32What is observability?
- 3:23The three pillars
- 5:46Performance and traces
- 6:56Why observability matters
- 8:53Observability challenges
- 10:26Complexity in kubernetes
- 12:38From monoliths to cloud
- 16:29Open source telemetry
- 19:34Open source toolkits
- 22:56Observability architecture
- 26:20Integrating security
- 30:05Production readiness
- 34:02Building ELK for production
- 37:22Prometheus architecture
- 38:15Tracing in production
- 40:21The cost of open source
- 42:01Demo: Logs.io platform
- 47:57Summary of benefits
- 49:19AI-driven troubleshooting
- 50:59Q&A session
related talks
13mArchitecting an open observability stack
11mKubernetes Observability
42mOpen Source for Better Observability
56mFull-stack observability and application monitoring with
37mBuilding Modern Observability Architecture With OpenSearch, OS Ingest...
14mObserving the Future
28mImplementing Observability in Kubernetes
11mBuilding lightweight observability system for Lean Teams