BM25 demystified #bbuzz
summary
Lucene and Elasticsearch change the default ranking from TF/IDF to BM25 and BM25 replaces unbounded term frequency with a k parameter that makes term-frequency effects saturate, and it applies length normalization via a b parameter using average document length. IDF still down-weights common terms (derived from Robertson–Sparck Jones), so stopwords impact scores less and the old coord hack becomes unnecessary. Parameters are tunable; behavior varies by corpus, so reindex or flip index similarity to compare.
AI-written summary · about
chapters (9)
- 0:00Introduction to BM25
- 2:10Why keyword search is hard
- 6:01Current default TFIDF explained
- 12:49Probability ranking principle
- 18:30From probabilistic to BM25
- 24:48Term frequency and eliteness
- 31:32BM25 parameters and saturation
- 34:19BM25 advantages over TFIDF
- 36:42Summary and resources
related talks
21mBeyond Significant Terms #bbuzz
37mLexical, Vector & Hybrid Search with Elasticsearch
47mThe Neural Search Frontier #bbuzz
13mA Beginner’s Guide to Semantic Search
45mHybrid search: Greater than the sum of its parts?
56mBM25 is So Yesterday: Modern Techniques for Better Search Relevance in Solr
01h 26mBagging to BERT- A tour of applied NLP
40mExploring reranking depth in modern search pipelines