Best Practices With Elasticsearch
summary
Practical guide to running Elasticsearch for large ETL and feature-generation pipelines: uses Spark + elasticsearch-hadoop connector and Python microservices, targets ~10 ms per query but hits 100M+ queries that take days without parallelism, relies on bulk API and reduced replicas/disabled refresh to speed initial ingest, sizes shards ~30–50 GB (avoid >500 GB), runs on i3 instances or EBS with heap <=30 GB, avoids nested/parent-child and nginx proxying, uses Curator + S3 snapshots for rollover and backups, and warns against Amazon’s managed Elasticsearch fork.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 1:16Who am I
- 2:27Agenda
- 3:17Elasticsearch 101
- 7:36Inverted Index
- 8:34Query DSL
- 9:10Terminology
- 13:41When should I consider Elasticsearch
- 16:48Most people use Elasticsearch
- 17:37Big data processing
- 18:29How it worked
- 19:55Machine Learning
- 20:46Feature Generation
- 21:39Feature Generation Example
- 22:42Query Generation Example
- 24:12Data Quality
- 24:57Architecture
- 26:10Databases
- 27:31What I Learned
- 29:23Lessons Learned
- 31:13Amazon Elastic Search
- 33:17Amazon Fort Elastic Search
- 34:33Sizing Your Cluster
- 38:04Spark
- 40:24Elastic Search
- 42:32Fast Reads
- 44:17Dont Abuse Refresh
- 50:05Curator
- 51:23Big Data Archive
- 52:04Split Brain
- 52:57Choose the Right Tool
- 53:54The Money Slide
related talks
29mHow (not) to Scale Elasticsearch for Data Analytics!
26mFrom Spark to Elasticsearch and Back
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
40mMaking your elastic cluster perform
44mSurviving Elasticsearch
24mOperating at scale with Elastic Search in production
37mBeyond the basics with Elasticsearch
54mScale Elasticsearch for Your Java Applications. Philipp Krenn. Java Fest 2020 Online