Automata Invasion
summary
Lucene integrates finite-state sets (FSA) and maps (FST/wFST) into queries, term dictionaries and analysis, delivering concrete wins: up to 50× speed or memory improvements, synonym dictionaries using an FST run 59× less RAM, Japanese analysis (Kuromoji) uses 12× less memory, autosuggest uses weighted FSTs and returns top suggestions efficiently, spellchecking queries directly from the index (no sidecar), and a compact on-disk/in-memory representation as one byte array that enables faster term lookups and a new MemoryPostingsFormat.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:57Who am I
- 1:33Automata
- 2:38Implementations
- 2:55FSA
- 3:34Map
- 4:48Analysis
- 5:22Realworld Applications
- 6:01Spellchecking
- 6:45Japanese
- 7:20Weights
- 8:27Synonyms
- 10:36Term State
- 11:17Inverted Index
- 12:28Conclusion
- 13:10Memory
- 14:12FST
related talks
40mApache Lucene: From Text Indexing to Artificial Intelligence
36mUnlocking Smarter Typeahead Search
31mIntelligent Japanese Search With OpenSearch
35mIn Memory Full Text Search with Kotlin
33mBeyond Benchmarks: Real-World AI Search Performance in OpenSearch (Lucene Vs Faiss Vs JVector)
01h 30mBuilding a Search Engine
09mSpell Correction & Query Segmentation
32mRetrieval Augmentation and Semantic Search at Scale