Deep Dive Fluent Bit: Logging & Stream Processing
summary
Live demo shows Fluent Bit ingesting a 1 million–line JSON log file, initially overwhelming Elasticsearch until a memory buffer limit throttles output and stabilizes indexing. New built-in stream processing on the edge runs SQL-like queries and windows in memory (no DB indexes), extracting ~60k matching records in seconds instead of minutes. The engine supports filtering, aggregation (20‑second tumbling windows with averages), chaining streams into tagged outputs, and plans for joins, timestamps backfill and Lua user functions.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:50Logging: basics
- 1:50Logging Handling
- 3:03Logging & Routing
- 4:33Fluent Bit v1.1 Major Changes
- 9:49Logging & Performance
- 19:07Stream Processing Capabilities
- 19:39DEMO #2: Edge computing v/s Cloud computing
related talks
40mStream Processing and New Approaches: Edge Processing
44mFluent Bit: Logging and Data Processing on the Edge
39mAdvanced Stream Processing on the Edge
31mLogging at Scale, Billion+ messages, 100K msg/sec
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
43mA Data Streaming Architecture with Apache Flink #bbuzz
24mHow To Remove Data Siloes: Tips & Considerations
38mKeynote: The Evolution of Data Infrastructure at Splunk