Search Engines · Ranking Algorithms
Why is BM25 the default ranking algorithm for search?
BM25 (Best Matching 25) powers Elasticsearch, Lucene, and SQLite. It replaces raw keyword counting with saturation ceilings and document length balances.
Repeated words hit diminishing returns
Going from zero mentions to three is a massive signal; repeating thirty more times barely adds to the score.
The k1 saturation knob
Engineers control this curve with a dial called k1. If k1 is low (around 1.2), the score hits its ceiling rapidly. If k1 is higher, repeated words retain influence longer before flattening out.
Don't reward long documents just for having more words
A large book mentions words simply because it is long; BM25 balances the count against average collection size.
The b length-penalty knob
A second dial named b controls how strictly length is penalized. At b = 1.0, long documents face full length penalization. At b = 0.75, the algorithm strikes a balance, letting comprehensive manuals compete without drowning out concise summaries.
Rare query terms still carry the decisive punch
Words present in nearly every file provide little signal; matching a rare term earns the bulk of the score.
The formula: Three simple filters multiplied together
By balancing term rarity, diminishing returns, and document length, BM25 delivers reliable results on millions of files in milliseconds.
Why BM25 still powers modern search engines
Even in the age of neural networks and vector embeddings, search engines like Elasticsearch, OpenSearch, and SQLite FTS use BM25 as their foundation. It requires zero training, never hallucinates, and finds exact technical codes, UUIDs, and function names faster than any AI model.
Predecessor: ELI5: TF-IDF · Longer version: Reference: BM25 · Essay: Build the Source of Truth
Sources: Stephen Robertson et al., Okapi at TREC-3 (1994); Christopher Manning et al., Introduction to Information Retrieval (2008).