Make Your Data FABulous
summary
Introduces the FAB theory (must choose two of Fast, Accurate, Big) and shows concrete Elasticsearch tradeoffs: term aggregations undercount when shard-level top-k truncation returns 60 vs expected 64 and can be fixed by raising shard_size (default ~size1.5+10) at the cost of work; unique-count uses HyperLogLog++ with a precision default 3000 (cap 40000) — 800 precision yields ~6% error, ~80KB gives ~1% error; relevance (BM25/TF‑IDF) skews across shards and is corrected with DFS query_then_fetch* or by using one shard.
AI-written summary · about
chapters (20)
- 0:00Start
- 1:38CAP Theorem
- 4:03Misconceptions Partition Tolerance is not a choice in a distributed system
- 7:00Robinson Crusoe
- 8:06dev/null breaks CAP: effect of write are always consistent, it's always available, and all replicas are consistent even during partitions.
- 9:12Mark Harwood
- 9:59Accurate Exact instead of approximate results
- 10:19Big Parallelization needed to handle the data
- 11:40Shard Unit of scale
- 19:30(Sub) Results Per Shard
- 20:49How Many? Results per shard Results for aggregation
- 24:09Cardinality Aggregation
- 30:56precision_threshold Default 3,000 Maximum 40,000
- 32:42Precompute Hashes? Client or mapper-murmura plugin
- 33:09It Depends large / high-cardinality fields low cardinality / numeric fields
- 33:44Improvement? "New cardinality estimation algorithms for HyperLogLog sketches"
- 35:12Term Frequency / Inverse Document Frequency (TF/IDF)
- 38:31DFS Query Then Fetch Distributed Frequency Search
- 40:41Single Shard Default in 7.0
- 43:01Conclusion
related talks
51mMake Your Data FABulous
33mMake Your Data FABulous
37mBeyond the basics with Elasticsearch
41mTips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond
29mReasoning about performance (in the context of search)
25mHigh on Cardinality, Low on Memory
48mAnalytics Architectures on Microsoft Fabric
41mBeyond the basics with Elasticsearch