Managing your Black Friday Logs
summary
Walks through concrete Elastic Stack practices for Black Friday traffic: use Beats → Logstash → Elasticsearch → Kibana, enable persistent queues or Kafka for buffering, use index templates and the rollover API to avoid daily-into-gazillion-shard growth, and reindex/shrink/split to reshape shards. Uses numbers from tests: 1,000 events/sec -> ~7 TB primaries, aim for 10–40 GB shard target, tune refresh/translog/indexing buffer and use sniffing or coordinating nodes, and finds a 3k–3.5k bulk sweet spot for throughput.
AI-written summary · about
chapters (35)
- 0:00Intro
- 0:37Agenda
- 1:24Elastic Stack
- 2:02Logstash
- 2:35Elasticsearch
- 4:04X-Pack will open -v6.3
- 4:54Elastic Cloud
- 5:38Elastic APM
- 5:52The Elastic Journey of an Event
- 9:37Terminology
- 10:00Partition
- 10:19Not all shards are created equal
- 13:01Shard Size
- 14:17Sizing exercise
- 15:41More about shard sizing
- 16:24Time-Based Data
- 17:24Templates
- 18:14Alias
- 18:52Rollover API
- 20:16Do not Overshard
- 21:05Recovering from gazillion shards scenario
- 22:51Cleaning up
- 23:22Shrink API
- 24:47Undersharded?
- 25:26Split API
- 26:05Scaling reads
- 27:01What is Bulk?
- 28:26What is the optimal bulk size?
- 28:39It depends...
- 29:43Avoid Bottlenecks
- 30:09Distributing the Load C
- 31:52Load Balancer
- 32:13Coordinating only Node
- 33:47Increasing write throughput - Knobs to turn
- 37:08We are hiring • Work with a disruptive technology
related talks
01h 01mManaging your Black Friday Logs
42mMake your logs work for you
35mHow to Survive Black Friday: Preparing e‑commerce for Peak Season
41mScaling Observability: From Chaos to Clarity with Open Source Tools
40mA needle in a logstack
29mWide Event Analytics
40mBuilding Observability with Elasticsearch: Practices for High Load
39mFast log management for your infrastructure