A New Approach to Logging as a Stack: Fluent Bit + PostgreSQL (FPS)
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Friday, 2020-11-20· at 17:55· Intrado Virtual Event Platform,·
summary
Fluent Bit outputs directly into PostgreSQL using JSONB storage and PL/pgSQL triggers to split and route logs into tables, with a default table to avoid data loss. The stack handles one terabyte of three years of Apache logs processed in two weeks, uses partitioned per month tables and indexes so many queries run under 1s, and includes Kubernetes ConfigMap deployment snippets, CockroachDB compatibility flag, and forthcoming SSL/schema support.
AI-written summary · about
chapters (14)
- 0:00Start
- 0:37Motivation
- 1:26Initial idea
- 2:09How to handle the data?
- 3:20PostgreSQL: JSONB
- 4:28Configuration
- 4:58Full list of options
- 5:28Query sample data
- 5:58Now with Apache logs
- 6:26Years of Apache Logs
- 7:19Let's deploy inside K8s
- 8:23Let's use some PL/SQL
- 11:17Query the data
- 11:34PL/SQL Function
related talks
34mFluentd: A Complete Logging Ecosystem for Kubernetes
25mPetabyte-scale Log management in Postgres using Bagger
27mUnder the Hood with Fluent Bit Operator: Kubernetes-native Log Processor
34mAnalytics and AI with PostgreSQL - Mission Possible!
34mPglog: a flexible log analyser
43mLogging Not Just for Lumberjacks
25mOne Size Does Not Fit All: A Polystore Architecture for Logs and Traces
09mLightning Talk: Fluentd at Scale; Keys to successful Logging Syndication