ElasticSearch: Big Data, Search and Analytics
summary
ElasticSearch stores huge datasets by treating each shard as the unit of scale (default five shards per index), moving shard files between nodes instead of reindexing, and using synchronous replicas for HA and read scaling. Design patterns include routing to force a user to one shard so a user’s queries hit a single shard, and time-based indices (daily/weekly/monthly) to retire or move old data. Facets like date histograms and terms provide fast, near real‑time aggregations; bulk update and smarter shard balancing are on the roadmap.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:27basics
- 2:22add another node
- 3:34#shards #nodes overallocation
- 6:27add replicas
- 7:41increase #replicas
- 8:10multiple indices
- 9:29sizing
- 11:49the "kagillion" shards problem (tm)
- 14:33"users" data flow
- 21:54"time" data flow
- 28:01the application
- 29:24histograms
- 30:41slice & dice
related talks
41mElasticsearch index management for paas style logging
53mBig Data: from mammoth to elephant
54mElasticSearch and Heka: How We Learned to Sieve an Elephant Through a Sieve
01h 00mELK Stack - All You Need for Search In Your Applications
35mHighly Available Search at Shopify
20mMigrating a Large Scale Search Dataset in Production in a Highly Available...
33mMake Your Data FABulous
29mBig Data Analysis