Advanced Stream Processing on the Edge
summary
An open source C tool brings stream processing on the edge to logs by parsing raw bytes into fields, running in-memory SQL-like queries and windowed aggregations, and emitting results instead of raw samples. It uses fluent-bit plugins (tail, CPU, Elasticsearch, InfluxDB, Slack webhooks), MessagePack serialization, single-threaded coroutines for nonblocking I/O, and Lua UDFs. Demo shows a 60s tumbling window computing CPU averages and posting alerts to Slack; project reports ~62 million deployments.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:10Logging
- 2:02Challenges
- 2:23Data Source
- 3:21Data Structure
- 5:33Flu Embed
- 6:48Data Pipeline
- 14:50Simple Messages
- 15:20Metadata
- 16:01Record
- 16:30JSON
- 17:44Daemon Set
- 18:19Stream Processing
- 19:53How Stream Processing Works
- 21:35Goal of Stream Processing
- 22:16Stream Processing in Fluent Bit
- 22:50Stream Processing in Data Pipeline
- 23:21Time Series Forecasting
- 28:57Internals
- 30:26Synchronous Operations
- 32:57Helpers
- 33:51Performance
related talks
40mStream Processing and New Approaches: Edge Processing
38mDeep Dive Fluent Bit: Logging & Stream Processing
34mStream Processing at the Edge with Apache Kafka
44mFluent Bit: Logging and Data Processing on the Edge
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
40mRobust Stream Processing with Apache Flink
24mStreaming analytics on edge with Kuiper and KubeEdge
48mTaming & Composing High Performance Stream-Oriented Processing Systems