Wings Stay On or Replacing On Site Search at Wikipedia
summary
Replaces Wikipedia’s on‑site search with an Elasticsearch-based index that handles ~870 million full‑text searches and 3.1 billion “find as” queries annually, keeps MySQL as the canonical store, and serves millions of queries per hour. Builds realtime indexing, an insource operator to search wikitext and rendered text, a suite of integration tests exercising Lucene/ICU/highlighting, and a production‑like dev environment via MediaWiki Vagrant. Rollouts use beta communities and phased deploys; plugins and load profiling fix performance and stability issues.
AI-written summary · about
chapters (11)
- 0:00Start
- 2:03Why We Chose Elasticsearch
- 6:27Beta Communities
- 10:27Stemming
- 11:02Transclusions
- 12:10Linking to a Youtube Video
- 13:17Render Text to the Page
- 19:43Unit Testing
- 24:57Load Testing in Production
- 25:51Mediawiki Vagrant
- 34:51Phased Rollout Approach
related talks
48mStateless Search: Decoupling Storage and Compute for Search Engines
35mThings I Learned the Hard Way Building a Search Engine
34mBuilding a WebScale Search Engine with a Data Driven Architecture
31mBring AI based Search to Your Website
35mYou've got questions. We've got answers! #bbuzz
06mStartup Showcase: WebKite
28mA complete Tweet index on Apache Lucene #bbuzz
31mExploring open data with BigQuery