Building a scalable logging platform
summary
A production logging pipeline built with fluentd/fluent-bit → Kafka → Elasticsearch/Kibana plus S3/Glacier for long‑term retention handles buffering, backpressure and multiple outputs; stack runs on Kubernetes with Dockerized shippers, JSON structured logs, and Graylog removed when Elasticsearch index management sufficed. It processes hundreds of GBs per month (a few TBs per quarter), archives to Glacier after 6–12 months, uses Sentry for immediate alerts, and reintroduces Kafka when the data team needs early raw access; the setup proves scalable and operational in real deployments.
AI-written summary · about
chapters (11)
- 0:00Introduction to scalable logging
- 5:43Build vs buy logging platforms
- 8:08Key requirements for logging systems
- 10:14Core components of logging architecture
- 13:27Our logging platform journey
- 15:36First iteration with Fluentd
- 18:04Experimenting with Kafka and alternatives
- 20:50Final architecture with Kubernetes
- 23:16Q&A session part one
- 31:25Q&A session part two
- 45:18Closing remarks
related talks
31mLogging at Scale, Billion+ messages, 100K msg/sec
34mFluentd: A Complete Logging Ecosystem for Kubernetes
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
34mFluentd: A Complete Logging Ecosystem for Kubernetes
09mLightning Talk: Fluentd at Scale; Keys to successful Logging Syndication
33mFluentd and Distributed Logging [I]
32mBuilding and scaling a log analytics platform - a serverless approach