Surviving Elasticsearch
summary
A production Elasticsearch deployment indexes 913 million documents (~120 TB on 82 servers) and handles ~30–40k QPS with ~3k docs/s; the architecture uses Kafka as the queue and source of truth, four parallel indices with alias swapping, and a stripped _source to reduce storage. Operational strategies include per-shard sizing to keep hot shards in cache, priority queues for controlled reindexing, reduced JVM heap to preserve OS page cache, request timeouts, local per-node logging clusters, and replayable Kafka consumers for failure recovery.
AI-written summary · about
chapters (5)
- 0:00Elasticsearch in marketplace
- 11:00Performance and indexing
- 21:52Kafka for data processing
- 26:15Cluster management
- 32:21Q&A
related talks
29mHow (not) to Scale Elasticsearch for Data Analytics!
24mOperating at scale with Elastic Search in production
01h 01mBest Practices With Elasticsearch
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
44mElasticsearch at Scale of Billions of Documents
40mBuilding Observability with Elasticsearch: Practices for High Load
57mBuilding a Scalable Analytical Solution Using IaaS and Elasticsearch
16mTop Elasticsearch tips