← Reference · Nestor G Pestelos Jr

Information Retrieval · Systems Architecture

Semantic Caching

Reference entry · last updated August 25, 2026

Semantic caching is a caching strategy that evaluates vector embedding similarity to serve previously generated responses for conceptually equivalent queries, rather than requiring exact string matches.[1] By mapping natural language requests into dense vector spaces, a semantic cache identifies when a new query shares intent with a cached entry, reducing redundant calls to foundation models.[2]

Operational mechanism

Traditional web caches use cryptographic hashes (such as SHA-256) of the request body. In natural language systems, two prompts with identical meaning (such as "What is the capital of France?" and "Tell me France's capital city") produce completely different hashes, causing cache misses.[1]

A semantic cache executes the following sequence:

1. Vectorization: The input query is passed through a lightweight embedding model to generate a dense semantic vector.
2. Similarity search: The vector is queried against an index of stored prompts using distance metrics (typically cosine similarity or Euclidean distance).[2]
3. Threshold evaluation: If the similarity score exceeds a configured threshold (for example, \( ext{sim} \ge 0.92\)), the stored completion is returned immediately.
4. Asynchronous insertion: On a cache miss, the request passes to the primary LLM, and the resulting prompt-response pair is embedded and indexed in the background.

The three-tier caching hierarchy

Production LLM architectures arrange caching in three progressive layers to balance speed, cost, and hit rates:[3]

• Exact match cache (Tier 1): An in-memory key-value store (such as Redis) keyed on prompt hash. Resolves in <5 milliseconds at zero embedding cost.
• Semantic vector cache (Tier 2): Evaluates cosine similarity over embedding spaces. Incurs 20–50 milliseconds of embedding latency, but captures rephrased queries.
• Proactive cache (Tier 3): Pre-computes and pre-warms responses for anticipated user questions or top-tier documentation items before users ask them.[3]

Stampede prevention

When multiple users ask identical or semantically equivalent questions simultaneously (such as during breaking news or a service outage), standard caches suffer from a cache stampede: every concurrent request misses the cache and triggers an expensive backend LLM call.[3]

Coalesce caching (or request collapsing) tracks in-flight requests. When a second identical request arrives while the first is still generating, the gateway blocks the second request and multiplexes the streamed response to both clients once received.[3]

Trade-offs and failure modes

Semantic caching introduces architectural risks that must be managed:

• Semantic drift / false hits: If the similarity threshold is set too low (e.g. <0.85), queries with subtle but critical differences (such as "How to install Python on Windows" vs "How to install Python on Linux") will return incorrect cached answers.[1]
• Temporal staleness: Caching time-sensitive answers (such as "What is the stock price of Apple?") yields stale outputs unless scoped with short Time-to-Live (TTL) policies or metadata invalidation tags.
• Embedding compute overhead: Generating embeddings for short queries incurs compute and latency costs that must remain significantly lower than direct LLM inference to justify the cache layer.

See also

• LLM Gateway
• Tokens
• Retrieval-augmented generation
• Semantic Caching (ELI5 explainer)

References

1. ↑ Bang, G., et al. "GPTCache: An Open-Source Semantic Cache for LLM Applications." arXiv preprint arXiv:2311.01723, 2023.
2. ↑ Redis Ltd. "Vector Similarity Search and Semantic Caching in Redis Enterprise," 2024.
3. ↑ Mitra, Sampriti. System Design for the LLM Era: Patterns and Principles for Production-Grade AI Architecture. Packt Publishing, 2026. Ch. 2: "Core Architectural Patterns for LLM System Design."