Solr Sparse Faceting #bbuzz
summary
Netarchive.dk indexes 500 TB into Solr and faces faceting on fields with hundreds of millions to billions of unique strings. The code changes claim about 2x speed on single-shard and 4x on multi-shard by reusing counter objects to cut GC, adding a sparse tracker to touch only updated terms, and looking up phase-one counts in Cloud merges to remove the “pit of pain.” For memory-bound cases an end-plane counting bitplane layout drops a 600M-term counter from >2 GB to ~300 MB while incurring modest CPU overhead.
AI-written summary · about
chapters (5)
- 0:00Start
- 0:06Solr sparse faceting
- 0:19Presentation tech level
- 8:21Sparse counting
- 18:19World Full Of Nothing
related talks
41mSearching large data sets in (near) constant time
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
42mFaster retrieval of top N documents in Apache Solr 8.6
39mAccelerating Aggregations with Lucene-Aware Execution in OpenSearch
44mScaling Apache Solr -- Features Of Resilience
21mThe Lucene Scalpel: Tuning Low-Level Segments for Multi-Lingual Concurrency
16mSolr Eclipse - Zomato's Journey to 64 Million Search Queries a Day
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...