Lightning Talk
summary
Claims a 60–70% cost reduction by changing log indexing from container to namespace level. Architecture uses fluent-bit DaemonSet + fluentd statefulset collecting container logs, fluentd aggregating and transforming, and AWS OpenSearch (behind an ALB) for storage. Instead of creating ~60 shards per namespace (one per container), logs are aggregated and sent as one index per namespace, cutting shards dramatically and keeping each data node under its ~1000-shard limit. Config tweak: remove container from logstash prefix so index = prefix_namespace.
AI-written summary · about
related talks
09mLightning Talk: Fluentd at Scale; Keys to successful Logging Syndication
22mPetabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From...
21mLightning Talks
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
11mLightning Talk: How Jenga & Tetris Can Help Run More Containers with Lesser Costs & Downtime
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
05mLightning Talk: Flexible Logging Pipelines with Fluentd and Kubernetes
01h 02mCentralised Logging without the blood, sweat and tears