Systems Design · AI
Semantic Caching
Remembering answers by meaning instead of exact words.
1. Old caches look at letters
Normal caches only reuse an answer if the words match letter for letter.
Why is this bad for AI?
Humans ask the same question in hundreds of different ways. Traditional caches miss almost every time, forcing the expensive AI model to run again.
2. Semantic caches look at meaning
A semantic cache turns questions into points on a map to see if they mean the same thing.
How does the map work?
The computer converts text into a list of numbers called an embedding. Sentences with similar meaning end up close together on the mathematical map.
3. Instant answers for zero dollars
When a match is close enough, the cache returns the stored answer immediately.
What if questions are slightly different?
Engineers set a threshold score. If questions are 92% similar, it reuses the answer. If lower, it asks the model to generate a fresh answer.
Longer version: Reference Entry on Semantic Caching
Sources: Bang et al., GPTCache (2023); Mitra, System Design for the LLM Era (2026).