Fluentd: Data Streams in Ruby World
summary
Fluentd is a Ruby open source log collector sold as a gem with 300+ plugins and a tag‑based routing core that connects inputs to outputs like Amazon S3 HDFS Elasticsearch Google BigQuery and Hadoop. It implements file buffer durability, retries and forwarding with load balancing and SSL auth, can do in‑stream aggregation (counts percentiles sums) and runs in production at LINE collecting billions of events per day, stable for years.
AI-written summary · about
chapters (12)
- 0:00Start
- 9:26read logs from file and write these on storages
- 12:06count events by string values
- 12:47aggregate numeric values
- 13:21various inputs: Linux performance (dstat)
- 14:57stream processing: external command
- 15:52stream processing w/ external server RPC
- 16:59Fluentd versions
- 18:21Fluentd on JRuby
- 19:04Fluentd on Windows
- 21:51Fluentd cluster statistics
- 24:24Fault-tolerance?
related talks
23mHow to become a data scientist with Ruby and Fluentd
31mBuild the Unified Logging Layer with Fluentd and Ruby
45mRuby for the soul of Big Data nerds
42m"Distributed stream processing in Ruby"
30mRuby in the Billions
36mPlugin-based software design with Ruby and RubyGems
45mmruby gateway for huge amount of realtime data processing
34mBuilding Data Driven Products with Ruby