Stream Processing and New Approaches: Edge Processing

on YouTube ↗40m

" summary

Fluent Bit adds a compact C stream processor that runs on edge devices, integrates into the Fluent Bit pipeline, and executes SQL-like queries, windowed aggregations and a basic linear regression forecast in-memory without indexing. Demo shows Elasticsearch choking on 1 million log lines while the embedded processor returns filtered streams immediately. The runtime is ~500KB, uses MessagePack, supports 50+ plugins and forward/HTTP outputs, employs coroutine-style synchronous IO, and is deployed >200,000 times daily.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 3:36Stream Processing
3. 16:42Demo
4. 29:25Time Series Forecast
5. 33:53Parsers

" related talks

• 39mAdvanced Stream Processing on the EdgeEduardo Silva · linux.conf.au 2020
• 38mDeep Dive Fluent Bit: Logging & Stream ProcessingEduardo Silva · KubeCon + CloudNativeCon Europe 2019
• 44mFluent Bit: Logging and Data Processing on the EdgeEduardo Silva, Masoud Koleini · KubeCon + CloudNativeCon North America 2019
• 21mSplunk Data Stream ProcessorJamie Grier · Flink Forward Global Virtual 2020
• 38mStreaming analytics and CEP ... #bbuzzTill Rohrmann, Fabian Hueske · Berlin Buzzwords 2016
• 39mDeclarative stream processing...Till Rohrmann, Fabian Hueske · Flink Forward Berlin 2016
• 34mStream Processing at the Edge with Apache KafkaJakub Scholz · DevConf.CZ 2023
• 24mStreaming analytics on edge with Kuiper and KubeEdgeRocky Jin · Kubernetes on Edge Day Europe 2021