A needle in a logstack
summary
Hands-on case study of an ELK pipeline that ingests mobile and backend logs, struggles with multi-line events and JVM memory tuning, and recovers actionable insights via dashboards. Pipeline uses Logstash + Elasticsearch + Kibana, plus lightweight shippers (filebeat/logstash-forwarder) and syslog; clusters run six nodes with ~200 GB each. Normal ingestion is ~15 GB/day (~300k events/10 min) but incidents spike to >10M and up to 30M events/10 min. Dashboards expose the exact server, bad stack trace loop, and the slow external iTunes calls so fixes are deployed.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:20ELK Stack family
- 2:42Logstash - deployment
- 3:07Logstash - problems we faced
- 12:20Elasticsearch - problems we faced
- 20:26Simple HIT/MISS statistics
- 20:58Varnish traffic per file type
- 29:55Response time analysis
related talks
01h 00mApplication Logging & Logstash
42mMake your logs work for you
01h 30mBehind the scenes of an ELK system
37mA Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
43mPractical Application Logging and Monitoring
39mFast log management for your infrastructure
41mScaling Observability: From Chaos to Clarity with Open Source Tools