Feeding Realworld Data into Prometheus Microservice Apps on Kubernetes
summary
A Kubernetes-based pipeline that converts factory signals on Raspberry Pi edge converters into OpenMetrics and ships them over MQTT with an MQTT-to-Prometheus bridge into per-tenant Prometheus + Grafana stacks, while duplicating raw time series into cassanda for long-term queries. A one-year prototype wires MQTT, Prometheus, Grafana, cassanda and TensorFlow, trains an LSTM on four metrics (CPU, disk write, net rx, cpu time) from a few days of data and shows rough predictions; deployment handles multi-tenancy by running separate Prometheus/Grafana instances per tenant.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:27OWL (Ostwestfalen Lippe)
- 0:52Smartfactory OWL
- 3:48OPC UA
- 4:33Edge Devices for Protcol Conversion
- 5:00Edge Device Raspberry PI (of course)
- 5:13How to do Continous Integration (for ARM)
- 6:26Howto do Remote Management
- 7:58by design Prometheus has
- 8:30Howto Transport
- 12:30Workflow at factory Floor
- 15:11Tensorflow Prototype
- 15:58Getting Data out of Prometheus
- 16:19Prediction
- 18:27References
related talks
34mBuilding and Scaling Robust Zero-code IoT Streaming Data Pipelines
34mStream Processing at the Edge with Apache Kafka
44mProcess and analyze millions of events per second with open source tools
11mBuilding Real-Time Feature Pipelines
25mFormula 1 telemetry processing using Kafka Streams
30mDeploy end to end ML pipeline using Apache spark ...
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
07mData Gravity and Kubernetes: Managing Large-Scale Data Ingest With Minimal Latency