Centralized Logging Patterns
summary
Compares four concrete patterns with demos and pitfalls: parse with filebeat → logstash (requires multiline rules and regex, demo shows 42 events vs 40 expected because stack traces were split), send directly via a log appender (no parsing but risks buffering loss and tight coupling if network fails), structured JSON files (write JSON, tail with beats, no parsing), and container/orchestration best practices (use a filebeat sidecar, persist registry, enrich with Docker/Kubernetes metadata and plan indices with index lifecycle management).
AI-written summary · about
chapters (19)
- 0:00Introduction and speaker background
- 2:06Problem with distributed logging
- 3:30Evolution of Elastic Stack
- 5:15Demo setup and Java logging
- 7:19Centralized logging architecture
- 9:52Viewing logs in Kibana
- 12:39Writing and debugging regex patterns
- 14:09Handling multi-line logs
- 16:41Structured logging directly to Logstash
- 18:50Downsides of direct structured logging
- 19:31Structured logging to JSON files
- 21:00Logging in containerized environments
- 24:31Filebeat registry file importance
- 26:16Filtering logs in containers
- 27:19Logging in Kubernetes
- 29:46Hot-warm-cold architecture for logs
- 31:57Index lifecycle management
- 34:45Future features in Elastic Stack
- 36:30Wrap-up and Q&A
