Fluentd Project Intro – Eduardo Silva & Masahiro Nakagawa, Treasure Data
summary
Fluentd explains a concrete logging pipeline: core+plugin architecture with inputs, filters, buffers and outputs, packaged as td-agent for easy installs and a container image for Kubernetes. It shows buffering strategies (local file chunks, chunk size and interval), retry and backpressure behavior, multi‑tier forwarding and common outputs (S3, HDFS, Kafka, Elasticsearch, MongoDB, Postgres). Demos and deployment notes claim production use at Treasure Data for multi‑thousand events/sec and successful fault‑tolerant delivery.
AI-written summary · about
chapters (8)
- 0:00Fluentd project intro
- 3:19Core features and plugins
- 8:46Data pipeline
- 14:03Streaming vs batch
- 20:11Performance and scalability
- 25:56Plugin development
- 32:05Back pressure and errors
- 35:07Q&A
related talks
34mFluentd: A Complete Logging Ecosystem for Kubernetes
35mIntro: Fluentd
33mFluentd and Distributed Logging [I]
37mFluentd: Cloud Native Logging
22mLogging: Fluentd & Fluent Bit
39mMaking Logs Work for you with Fluentd
31mBuild the Unified Logging Layer with Fluentd and Ruby
46mBuilding a scalable logging platform