Kafka Monitoring: What Matters!
summary
Describes concrete monitoring: export JMX → Prometheus → Grafana for broker metrics and use Burrow lag detection for consumer offsets; alert on four definite metrics — active controller count (must be 1), offline partition count (0), min ISR breaches, and under‑replicated partitions — and track per‑partition fetcher lag and log flush latency. Converts offset lag to time (example: 20 minutes behind) rather than absolute counts, cites scaling fixes (add partitions, add consumers, scale servers) and a soft limit ~4000 partitions per broker.
AI-written summary · about
chapters (12)
- 0:00Introduction to Kafka monitoring
- 8:26Kafka core fundamentals
- 10:52Beyond monitoring to observability
- 14:23Key Kafka performance areas
- 18:40Producer and broker metrics
- 23:13Consumer offsets and lag
- 25:18Consumer lag with Burrow
- 30:19Time-based consumer lag analysis
- 32:43Trend analysis and predictions
- 34:26Infrastructure and limits
- 35:31Resources and references
- 36:00Q&A and closing
related talks
33mObservability-First Kafka: Engineering Visibility at Scale
23mWhen Kafka Goes Cloud Native: Observability That Actually Works!
06mIGNITE: Observability with Grafana & Prometheus for Kafka on Kubernetes (CFK)
52mMonitoring Kafka without instrumentation using eBPF
32mImproving Kafka monitoring with native Prometheu...
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
27mObserving Failure in Reactive Systems
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus