Dynamic Broker-Side Filtering for Kafka
summary
Claims a broker-side filtering implementation that closes a 7‑year KIP gap by shipping a Kafka‑compatible filtering layer (native Java/Python SDKs, control plane over a compacted topic, and a scalable data‑plane of filter functions) that emits one output topic per selector. Benchmarks report sub-millisecond filtering with p99 under 25 ms and production deployments in financial services and logistics showing 60–80% network reduction. Demo uses StreamNative’s Kafka‑on‑Pulsar stack, works with live queries despite a brief UI hiccup, and is deployable as a drop‑in Kafka consumer replacement.
AI-written summary · about
chapters (16)
- 0:00Introduction to broker-side filtering
- 2:53Problem with Kafka's default behavior
- 5:44Why Kafka lacks broker-side filtering
- 7:09Solution architecture overview
- 8:10Native Kafka SDK implementation
- 10:17How filtering works step-by-step
- 11:19Advantages of per-filter topics
- 12:43Orchestrator design and Kubernetes pattern
- 14:27Scaling and parallel filtering
- 16:34Demo setup and architecture
- 22:09Live demo challenges and recovery
- 23:14Demo results and latency metrics
- 31:07Dynamic scaling and partition adjustments
- 33:36Filtering performance analysis
- 36:06Future roadmap and next steps
- 37:48Closing remarks and Q&A
related talks
54mMicroservices with Kafka
26mProduction Grade Kafka on K8s
36mKafka Without Disks: Making Streaming Brokers Disposable on K8S
13mScaling Kafka on Kubernetes
27mRunning Kafka in Kubernetes a practical guide
06mPREVIEW: Real-time Stream Processing at Scale
01h 02mModern Event-Driven Workloads with Knative
14mFast Data on-Ramp with Apache Pulsar on K8