Searching large data sets in (near) constant time
summary
Claims a generic Solr-side system that limits expensive searches to a 60k cap and a 400–500 ms budget by estimating hit counts from tiny index buckets, using an index-time random bucket field, local-shard sampling, and quantile math to pick the cutoff. Built components include an estimated-hit-count plugin, T-Digest quantiles and HyperLogLog++ sketches; deployment halves average CPU headroom issues (from 25% to 50% utilization target) and cuts typical heavy-query result sets by ~80%, dropping median latency from ~1.2s to ~450ms.
AI-written summary · about
chapters (9)
- 0:00Project context and goals
- 2:16Initial strategy for capping
- 6:14The proposed technical plan
- 8:12Sampling challenges in Solr
- 11:10Statistical approach to sampling
- 16:21Implementation details
- 23:03Results and performance
- 26:27Addressing facets and conclusions
- 31:12Q&A session
related talks
29mSolr Sparse Faceting #bbuzz
42mFaster retrieval of top N documents in Apache Solr 8.6
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...
39mAccelerating Aggregations with Lucene-Aware Execution in OpenSearch
16mSolr Eclipse - Zomato's Journey to 64 Million Search Queries a Day
44mScaling Apache Solr -- Features Of Resilience
19mEnhance Search Performance At Scale with
31mMaximize Resource Efficiency with Separated Index and Search Workloads