Lightning Talk: Scaling the Fluent Bit Kubernetes Filter in...
Lightningat FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd Europe 2021 · on Tuesday, 2021-05-04·
summary
Fluent Bit’s Kubernetes filter is changed to fetch pod metadata from the node kubelet instead of the API server, implemented as a kubelet metadata option. On a 2,000-node cluster with ~30,000 pods, Fluentd-style behavior generates ~60–70k list/get calls over three hours, causes p99 list latency spikes, kubectl hangs of 15–60s and ~1.5 GB agent memory; enabling the kubelet option drops Fluent Bit’s API list calls to zero, cutting control-plane load and reducing memory and latency pressure.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:32Overview
- 1:03Problem
- 1:50Architecture
- 4:33P99
- 5:11List Latency
- 5:25Customers
- 6:01Prometheus
- 7:57Fluent Bit with Kubernetes
related talks
05mLightning Talk: Flexible Logging Pipelines with Fluentd and Kubernetes
27mUnder the Hood with Fluent Bit Operator: Kubernetes-native Log Processor
05mLightning Talk: What I Wish I’d Known about Fluentd with Kubernetes - Bryan Boreham, Weaveworks
34mFluentd: A Complete Logging Ecosystem for Kubernetes
34mPower of O11y with FLUENT BIT in AWS EKS environment
10mLightning Talk
34mPanel: Improving and Managing Kubernetes at Scale
35mIntro: Fluentd