Using Apache Pulsar to provide real-time analytics on the edge
summary
Claims a working pattern: deploy trained models as PMML into Apache Pulsar Functions on IoT gateways, subscribe to MQTT sensor topics, keep the latest feature vector in a hash map, run the model locally and publish alerts when the prediction crosses a threshold (example uses remaining service hours < 24). Uses the state store to push model bytes, packages dependencies as a NAR via Maven, supports Java Python and Go, and runs on Raspberry‑Pi–class gateways with configurable parallelism.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:27Real-Time Machine Learning on the Edge
- 3:53Edge Computing Topology
- 5:33Logical Layers
- 7:40Apache Pulsar's Multi-tiered Architecture
- 8:20Edge Computing with Pulsar
- 9:36Benefits of Running Pulsar Broker on the Edge
- 11:12Pulsar Functions
- 12:36Pulsar Function API
- 13:26Pulsar Function Context Object
- 14:25Pulsar Function - Third Party Library Support
- 21:45Model Execution
- 25:08Function Deployment
- 27:07Function Configuration
- 29:15Summary
related talks
36mMachine Learning with Apache Beam
19mModern Data Science in a Cloud Native World
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
34mReal-Time Analytics: Going Beyond Stream Processing with Apache Pinot
43mHail Hydrate! From Stream to Lake Using Open Source
42mBuilding a near real time personalized feed using data streaming and Aerospike
40mStreaming Geospatial Vector Data with Apache Projects
34mML and analytics addressing nationwide COVID-19 impact and recovery