← ELI5 · Nestor G Pestelos Jr

Information Retrieval · Term Weighting

How do search engines know which words actually matter?

The classic answer is TF-IDF. It balances how often a word appears on a page against how rare that word is across the whole library.

Step 1: Count how often the word appears locally (TF)

A document showing Term Frequency: counting repeated words on a single page ARTICLE: COFFEE BREWING The best way to brew coffee is... Hot water extracts coffee oils... Serve fresh coffee immediately... TERM FREQUENCY (TF) "coffee" appears 3 times High Local Count

Term Frequency measures local concentration: a page about coffee will naturally repeat the word coffee.

Why local counting alone fails

Words like "the", "and", and "is" appear even more frequently than "coffee" on the same page. If a search engine only counted words, every search result would be dominated by common grammatical filler.

Step 2: Penalize words that appear everywhere (IDF)

Comparing common words across all books against rare words present in few books WORD: "THE" Appears in 10,000 of 10,000 books IDF Score: Near Zero Zero Rarity Signal WORD: "ESPRESSO" Appears in 4 of 10,000 books IDF Score: Huge Multiplier High Rarity Signal

Inverse Document Frequency measures rarity: words that appear in every file get discounted, while rare words get boosted.

How the math suppresses common words

IDF takes the total document count and divides it by the number of documents containing the term. By wrapping that ratio in a logarithm, widespread words approach zero, acting as an automatic noise filter without needing manual ban lists.

Step 3: Multiply them together to find true keywords

A 2x2 grid showing how TF and IDF combine to distinguish real topics from noise HIGH COUNT · COMMON WORD "the", "and", "is" (Filler) TF-IDF: Low Score HIGH COUNT · RARE WORD "espresso", "roast" (True Topic) TF-IDF: Highest Score LOW COUNT · COMMON WORD "maybe", "also" (Background) TF-IDF: Negligible LOW COUNT · RARE WORD "cappuccino" (Passing Mention) TF-IDF: Moderate Score

Multiplying local frequency by global rarity gives the highest score to words that are frequent on this page but rare everywhere else.

Where TF-IDF stops: Length bias and keyword stuffing

Showing two flaws in TF-IDF: runaway scores from spam repetition, and bias toward giant books FLAW 1: UNBOUNDED TF Repeating a word 100 times gives 100 times the score. Vulnerable to keyword stuffing FLAW 2: LENGTH BIAS A 500-page book mentions words purely by accident. Crushes short, focused summaries

Because TF-IDF scales linearly and ignores document length, modern search engines evolved to use BM25 instead.

How BM25 repaired these two flaws

In the 1990s, researchers introduced BM25. It capped term frequency so repeated words hit diminishing returns, and normalized for document length so long encyclopedias stopped crushing concise articles.

Next evolution: ELI5: BM25 · Reference: BM25 · Essay: Build the Source of Truth

Sources: Karen Spärck Jones, A Statistical Interpretation of Term Specificity (1972); Gerard Salton and Michael J. McGill, Introduction to Modern Information Retrieval (1983).

↑ Back to top