Measuring Search Engine Quality using Spark and Python
summary
A system built with Python and Apache Spark replays a reference set of 80,000+ user queries against Apache Solr configurations and computes Average Click Rank (ACR) as a numeric proxy for user satisfaction. It actually replays click logs, computes per‑query click ranks, reports average shifts and distributions, isolates slow queries, and uses multiprocessing + S3/EC2 to scale. The pipeline identifies which configurations measurably improve ACR so only those go to slow, risky A/B tests.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:03Search engine migration context
- 3:49A/B testing limitations
- 5:14Offline evaluation system
- 8:26Generating search results
- 11:19Spark for click rank
- 12:27Click data to canonical format
- 14:36Processing search results
- 15:37Computing click rank
- 16:41Visualizing results
- 17:43Solr production outcome
- 18:26System additional uses
- 19:56Future work and learning
- 21:04Q&A
related talks
20mMeasuring Data with OpenSearch to Better Serve the Community
26mFrom Spark to Elasticsearch and Back
39mMastering OpenSearch Benchmark: Advanced Techniques for Power Users
57mUnlocking the Power of Hybrid Search: A Deep Dive into Python-Powered Precision
28mA Deep Dive into Python-Powered Precision and Scalability
29mFrom Java to Python: Migrating Search Functionality at billiger.de
41mPainless / Painful? A Custom Ranking System in Elasticsearch
28mSuper Search with OpenSearch and Python