When Kafka Goes Cloud Native: Observability That Actually Works!
summary
Enterprise Kafka on Kubernetes gets vendor-agnostic observability using OpenTelemetry collectors deployed as OpenTelemetry collectors as DaemonSets, routing metrics to Prometheus, traces/logs to OpenSearch + Grafana, and keeping SLOs (six objectives, zero breaches, 100% error budget) as proof of reliability. Operational wins include reduced investigations from 2h to 10m; hard constraints to plan for are cardinality explosion, semantic convention mismatch, collector resource contention, and metric naming drift. Rollout blueprint: assess, pilot (one cluster), instrument, visualize, then scale.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:23What is event streaming
- 2:04Why Kubernetes for Kafka
- 4:12Observability in Kafka
- 6:59Observability foundations
- 7:41OpenTelemetry pipeline
- 9:48Legacy observability limits
- 11:55Kafka metrics structure
- 13:42Kubernetes infrastructure
- 14:45Client metrics
- 15:48Reliability with SLOs
- 18:22OpenTelemetry migration limits
- 20:47OpenTelemetry adoption blueprint
- 22:55Key takeaways
related talks
33mObservability-First Kafka: Engineering Visibility at Scale
14mObserving the Future
29mKafka on Kubernetes: Does it really have to be the hard way?
40mKafka, DevOps...And Resilience for All
01h 02mDesigning an open-source observability stack for Kubernetes
28mImplementing Observability in Kubernetes
33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]
30mKafka on Kubernetes: From Evaluation to Production at