Managing your Black Friday Logs
summary
Uses the Elastic Stack (Beats → Logstash/ingest → Elasticsearch → Kibana) to survive Black Friday traffic spikes, sizing around 45 GB shards and planning for ~7 TB of indexed data split into roughly 100 shards on ~16 hosts. Implements time-based indices with rollover, sets refresh_interval to 60s, and tunes bulk ingestion (~10 MB batches) and disk I/O. During the sale the cluster increases replica count to absorb search load and reduces replicas back after the event; this approach proves effective.
AI-written summary · about
chapters (14)
- 0:00Managing Black Friday logs
- 1:31Elastic stack overview
- 3:25Real-time log collection
- 10:12Log data transformation
- 17:53Shard management and replication
- 23:47Time-based indexing
- 30:55Index aliases and rollover
- 36:45Query optimization
- 43:17Elasticsearch coordination
- 47:57Segment refresh tuning
- 52:38Log transactions and persistence
- 57:50Auto-scaling in cloud
- 59:19Advanced pipelines
- 1:00:24Conclusion
related talks
38mManaging your Black Friday Logs
35mHow to Survive Black Friday: Preparing e‑commerce for Peak Season
42mMake your logs work for you
41mScaling Observability: From Chaos to Clarity with Open Source Tools
40mA needle in a logstack
44mSurviving Elasticsearch
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
29mWide Event Analytics