Petabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From...
summary
Fluentd and Fluent Bit run as a node-level daemonset that tails container stdout, buffers per-container, compresses batches to gzip, and uploads 10s objects to S3; a separate metadata service writes container metadata to S3 and a Fargate reader joins logs+metadata and forwards to the central log store over AWS Transit Gateway. Result: 30,000 events/sec per node, 90,000 events/sec total in tests, median latency ~9.8s (99th <14s) and 92% cost reduction (~$50k saved per PB).
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:30What is Fluentd
- 1:46Challenges for Scale Logging
- 3:10Fluentd Fluent Bit
- 6:25Architecture Patterns
- 8:07Use Case
- 8:37Log Transport
- 11:29Challenges
- 13:20Architecture
- 15:41Offloading
- 18:00Cost Breakdown
- 19:29Test Cluster
- 20:06Test Results
- 20:46Summary
related talks
42mLogging for Containers
22mLogging: Fluentd & Fluent Bit
34mGetting Started With Logging in Kubernetes
31mLogging at Scale, Billion+ messages, 100K msg/sec
46mBuilding a scalable logging platform
22mFluent Bit - Swiss Army Tool of Observability Data Ingestion
34mFluentd and Fluent Bit
33mFluentd and Distributed Logging [I]