Realtime Analytics Geofencing Kafka
summary
POC demonstrates real‑time geofencing for a steel company (80 barges reporting positions every 15 minutes) using Kafka topics, user‑defined Java geo functions and spatial indexing by geohash. Prototype with ksqlDB works functionally but forces heavy aggregation, loses per‑vehicle ordering when repartitioning, and struggles to scale. Rewriting in Kafka Streams uses a global KTable keyed by geohash, keeps joins partition‑local, and performs reliably; optional external service Tile38 can replace in‑process geometry checks. Overall: ksqlDB for quick prototyping, Kafka Streams for production.
AI-written summary · about
chapters (14)
- 0:00Introduction to geofencing
- 2:57Geofencing use cases
- 5:05Customer project background
- 7:50Geospatial tools and concepts
- 9:35Overview of Kafka ecosystem
- 17:38Geofencing architecture
- 19:49Geofencing with ksqlDB
- 26:52Initial ksqlDB join attempts
- 32:36Geohashing for scalability
- 41:01KsqlDB flattening challenges
- 45:22Message ordering issues
- 48:22Kafka Streams implementation
- 55:20Alternative: Tile38
- 56:54Conclusion and resources
related talks
42mLocation Analytics - Real-Time Geofencing using Kafka
31mFun with Sensors
57mTaming IoT Data: Making Sense of Sensors with SQL Streaming
39mKafka as a Platform: The Ecosystem from the Ground Up
41mKafka as a Platform: The Ecosystem from the Ground Up
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
08mPREVIEW: Kafka Stream Processing for Everyone with KSQL