Make Your Data FABulous
summary
Explains concrete trade-offs and concrete fixes: Elasticsearch shards cause a top-term aggregation to return 56 not 64 because per-shard top-k defaults and routing skew; raising shard_size fixes it by fetching more per-shard results. Describes the FAB theory (fast accurate big) as a choice of two, details Murmur3 hashing and routed shards, uses HyperLogLog++ for cardinality with precision-memory trade-offs, and shows DFS distributed frequency search costs and when to accept approximation versus exact counts.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:52What is the perfect datastore solution?
- 1:26Pick your tradeoffs
- 1:40CAP Theorem
- 4:45Partition Tolerance is not a choice in a distributed system
- 5:38Consistency in ACID is a predicate Consistency in CAP is a linear order
- 6:30Robinson Crusoe
- 7:53FAB Theory
- 12:10Terms Aggregation
- 23:45Cardinality Aggregation
- 23:55Naive implementation: Hash set Predictable storage and performance?
- 26:45precision_threshold Default 3,000 Maximum 40,000
- 27:02Memory precision_threshold x 8 bytes
- 30:46Term Frequency / Inverse Document Frequency (TF/IDF)
- 31:55Inverse Document Frequency num Docs
related talks
45mProfessional Data: Using Elasticsearch's Aggregation Framework
43mBeyond Shuffling: Scaling Apache Spark
38mAdd ALL the things: abstract algebra meets analytics
42mMake Sense of your (BIG) data!
31mScaling Facets to the Stars 🌟
48mDeeper Data Dimensions with Kibana
33mRelevant Data Analysis: Apache Solr Analytics #bbuzz
10mTurning data into leverage