Streamlining Search Quality: Search Relevance Workbench
summary
Search Relevance Workbench (SRW) is an open source, engine-agnostic suite that collects click streams using the User Behavior Insights standard, ingests labeled judgments or LLM/implicit clicks, and runs online/offline evaluations with metrics like Jaccard, RBO and NDCG 0.35 on a 150-query sample. Demo shows a simple title boost that helps three handpicked queries (NDCG ≈ 0.78) but only slightly improves the full sample, visual comparison of result churn, APIs for automation, and dashboards shipping in OpenSearch 3.2.
AI-written summary · about
chapters (9)
- 0:00Introduction and relevance
- 1:45Measurement and tooling
- 4:12Philosophy of the workbench
- 5:54Core workbench concepts
- 8:35Demo for the freshman user
- 15:23Query set comparison
- 22:04Systematic search evaluation
- 29:34Advanced analytics for users
- 35:18Future work and conclusions
related talks
43mThe life of a search engine administrator
44mBuilding a Modern Search Engine: The Road Ahead
18mZero To Hero: Building a Developer-First Search Experience With OpenSearch
55mHow to Reverse Engineer the SERPs to Find The Real Reasons Why You’re Not Ranking
45mBuilding amazing searches with Search API
39mSPONSOR WORKSHOP / Seth Larson / Elastic
40mBuilding a search platform 101 #bbuzz
11mRethinking Dataset Discovery with DataScout