Monitoring Kafka without instrumentation using eBPF
summary
eBPF captures Kafka protocol events in the kernel and Pixie parses them to produce real metrics without changing app code: deploy a Pixie operator per Kubernetes node and it subscribes to socket events, filters Kafka control frames, and parses produce/fetch/offset commits. It surfaces measure consumer lag in seconds without external components, detect producers, consumers, brokers and per-topic throughput, shows flamegraphs and rebalance durations, and claims far lower overhead than JMX exporters (JMX ~10% CPU).
AI-written summary · about
chapters (16)
- 0:00Introduction
- 0:50Kafka
- 4:27What is eBPF
- 7:30bcc
- 8:54bpftrace
- 10:48eBPF exporter
- 11:47Introducing pixi
- 13:29How it works
- 16:19Challenges
- 18:08Demo
- 19:47Pixi
- 24:22Identify Kafka clients
- 29:05Kafka consumer luck
- 32:43Kafka consumer latency
- 38:34Kafka consumer rebalancing
- 44:54Questions
related talks
31mZero instrumentation observability with eBPF
37mZero-instrumentation observability based on eBPF
28mBpftrace Meets Pixie: Dynamic Monitoring of K8s Clusters Unleashed
25mExpanding eBPF’s Reach: From Batteries-Included Auto-Instrumentation To E2E Observability Pipelines
26mTriaging Real-time Security Threats with eBPF-powered Observability
04mInstant, Auto-telemetry for K8s Observability
17meBPF in Microservices Observability
42mMonitoring Kubernetes Traffic with eBPF