Build the Unified Logging Layer with Fluentd and Ruby

on YouTube ↗31m

" summary

Fluentd is a Ruby data collector that actually connects many inputs to many outputs and ships reliably using configurable buffering to disk or memory The talk shows concrete deployments including one operator running 2,500 servers with one aggregator per ~100 hosts and a copy plugin wiring data to Elasticsearch and Hadoop Implementation details matter: the tail input contains ~700 lines of inode and rotation logic, the ecosystem has ~300 plugins and ~200 outputs, and a new filter API and Windows agent are in progress

AI-written summary · about

" chapters (24)

1. 0:00Introduction
2. 1:54Who am I
3. 2:44What is Fluentd
4. 4:08Data Collection
5. 4:59Extensible
6. 7:45MetaReliability
7. 8:55Use Cases
8. 15:05Architecture
9. 15:15InputOutput
10. 15:55Intel
11. 16:47TailInput
12. 18:09TCPInput
13. 18:34Parser
14. 19:19Parser plugin
15. 20:09Regular expression parser
16. 20:27Buffer
17. 23:00Plugin
18. 23:51Output Plugins
19. 24:39Single Value for Matter
20. 25:39New Filter Plugin
21. 27:55Fluentd Package
22. 28:39Treasure Data Package
23. 29:09Fluentd
24. 30:10Outro

" related talks

• 28mFluentd: Data Streams in Ruby WorldSatoshi "Moris" Tagomori · Red Dot Ruby Conference 2014
• 39mMaking Logs Work for you with FluentdPhil Wilkins · Conf42 Cloud Native 2021
• 37mFluentd: Cloud Native LoggingMasahiro Nakagawa, Yuta Iwama · KubeCon + CloudNativeCon North America 2019
• 34mFluentd: A Complete Logging Ecosystem for KubernetesMasahiro Nakagawa, Yuta Iwama · KubeCon + CloudNativeCon China 2019
• 23mHow to become a data scientist with Ruby and FluentdKiyoto Tamura · GoGaRuCo 2014
• 22mLogging: Fluentd & Fluent BitEduardo Silva, Masahiro Nakagawa · KubeCon + CloudNativeCon North America 2020 Virtual
• 35mIntro: FluentdEduardo Silva, Masahiro Nakagawa · KubeCon + CloudNativeCon China 2018
• 33mFluentd and Distributed Logging [I]Masahiro Nakagawa · KubeCon + CloudNativeCon North America 2017