Multilingual Search System - How to Build and Improve!?
summary
Company handling ~287 billion emails across >40 PB builds a pipeline using Apache Lucene/Elasticsearch, a logistic regression language detector trained on Wikipedia to tag language with high English recall, and a custom analyzer that applies a hyphenation compound word token filter feeding into the German analyzer to split compounds and normalize umlauts. Documents live in a separate index per language. Query-log analysis plus storing subwords in a separate field and boosting the original content field restores precision while increasing recall across ~3 million weekly searches.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:45Company Introduction
- 1:40Company Story
- 2:49Tokenizer
- 4:15Character Equivalence
- 5:01Custom Analyzer
- 6:43Example
- 7:51Language Detection Model
- 10:20Separate Index
- 11:05Separate Field for Language
- 12:57Language Detection
- 13:10Search
- 13:54Balance
- 15:25Rewrite Queries
- 16:46Build Language Detection
- 17:44Open Source
related talks
36mTuning Elasticsearch for English-Language Precision
46mFull-Text Search Internals with Elasticsearch
37mAlgorithms & data-structures that power Lucene & ElasticSearch
39mEntity Linking at scale with Lucene
01h 28mBuilding a Semantic Search Engine
38mIntroduction to Search
42mBuilding a real-time news search engine #bbuzz
43mSearch Logs + Machine Learning = Auto-Tagged Inventory