Fluent Bit: Logging and Data Processing on the Edge
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 16:25· Pacific Ballroom, Salon 23-24 - Marriott Marquis San Diego Marina Hotel·
summary
Fluent Bit is a C-based, MessagePack-backed log agent with a pluggable input/filter/output pipeline and an added stream processing on the edge layer that runs SQL-like queries over hopping/tumbling windows (demo: 30s hopping window) to compute averages, run a simple linear forecast (memory next 100s) and emit an alert when forecast >1.5GB to Slack. It supports snapshot on trigger (keep last 5s or N records, flush on 500) and ships as v1.3.3 with 100+ contributors and ~50 million deployments; v1.4 plans include config validation and expanded time-series functions.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:54Data is everywhere
- 1:38Challenges
- 6:13Get more context
- 7:21Message Back
- 8:08Global Architecture
- 8:58Deploying Fluent Bit
- 10:20Project Updates
- 11:07Happy Stats
- 11:57AWS
- 13:55Fluent Bit 133
- 15:57Stream Processing
- 17:28Streams of Data
- 18:54Fluid Configuration
- 22:34Time Series
- 23:56Forecasting
- 25:14Forecasting configuration
- 31:11Snapshot
- 31:51Flush Snapshot
- 32:51Demo
- 35:14Questions
related talks
40mStream Processing and New Approaches: Edge Processing
38mDeep Dive Fluent Bit: Logging & Stream Processing
39mAdvanced Stream Processing on the Edge
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
31mLogging at Scale, Billion+ messages, 100K msg/sec
43mA Data Streaming Architecture with Apache Flink #bbuzz
28mSweet Streams Are Made Of These
41mBeyond the Hype: When Apache Flink Solves Real Problems