Tips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
summary
Shows concrete fixes and measurable wins: change integer fields to keyword IDs reduces p99 by ~20 ms on term-heavy filters; avoid date math and rewrite date ranges (or invert broad ranges) to restore caching and shave ~10–15 ms p99; build feature-focused indices so request-cache hit rises from ~6% to ~42% for targeted aggregations. Uses a Kafka→Elasticsearch replay tool (CAT) for tests, runs three ~160‑node clusters plus an offline cluster, reaches ~1,000,000 RPM, ~360M docs, ~120–150 ms p99 and ~5,000 updates/s.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:10Current Elasticsearch scale
- 6:36Store IDs as keywords
- 11:19Avoid date math in filters
- 16:26Feature-focused indices
- 24:58Scale a year later
- 29:14Remaining challenges
- 30:58Q&A
related talks
29mHow (not) to Scale Elasticsearch for Data Analytics!
01h 01mBest Practices With Elasticsearch
37mBeyond the basics with Elasticsearch
44mSurviving Elasticsearch
16mTop Elasticsearch tips
38mAccelerating Query Performance in CrateDB
40mMaking your elastic cluster perform
43mCattle vs. Pets - A Cost-Effective Elasticsearch Architecture to Scale-Out...