Streaming Geospatial Vector Data with Apache Projects
summary
Live AIS demo shows a working pipeline that ingests satellite feeds, converts messages to simple features, and streams updates to an in-memory map via Apache Kafka while persisting history to HBase/Accumulo and Parquet on S3. GeoMesa provides a Kryo/Avro message format, command-line tools and NiFi processors; flows convert TCP/HTTP -> GeoAvro -> Kafka/HBase and write duplicates to S3. The in-memory store uses a feature-id hashmap plus coarse grid buckets; analytics run with Storm and KSQLDB experiments to answer 30-day tracks, port departures and closest-approach queries.
AI-written summary · about
chapters (11)
- 0:00Introduction to GeoMesa
- 1:52GeoMesa architecture overview
- 5:23Live AIS data demo
- 8:52Kafka for streaming geospatial
- 10:17Apache Kafka deep dive
- 12:46In-memory spatial database
- 17:37Apache NiFi for ETL
- 23:32NiFi geospatial processors
- 24:35Streaming analytics with Kafka
- 26:42KSQL spatial extensions
- 27:03Q&A and future plans
related talks
43mHail Hydrate! From Stream to Lake Using Open Source
36mBuilding a real time...
50mA Journey in Space with Apache Kafka data streams from NASA
43mBuilding a community fountain around your data stream
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
38mOn Track with Apache Kafka
22mApache SIS library for geospatial applications
49mStreaming Visualisation