How (not) to Scale Elasticsearch for Data Analytics!
summary
Managed Elasticsearch runs on 500+ JVM instances holding ~4 PB and handling ~20+ billion ingests/day (peaks ~50M sources/day); scaling techniques include bulk writes with _routing per-shard, raising index.refresh_interval, and setting translog durability to async to boost ingestion, plus date-based indices and vigilant mapping to avoid field explosion. Built tools include query logger, gatekeeper, and query optimizer that cut search latency ~25%, saved ~20% disk and ~30% search latency from index tuning, and replaced a $1M licensing cost with a custom security plugin.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 3:32Search platform at PayPal
- 8:54Elasticsearch basics
- 12:42How indexing works
- 15:09Scaling Elasticsearch overview
- 16:55Low ingestion throughput fixes
- 22:08Mapping explosion and sparse fields
- 24:33Expensive query problems
- 28:05Oversized shards issues
- 28:47Q&A
related talks
01h 01mBest Practices With Elasticsearch
24mOperating at scale with Elastic Search in production
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
57mBuilding a Scalable Analytical Solution Using IaaS and Elasticsearch
44mSurviving Elasticsearch
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
44mElasticsearch at Scale of Billions of Documents
37mBeyond the basics with Elasticsearch