Systems Architecture · Artificial Intelligence

Prompt Caching

Reference entry · last updated August 26, 2026

Prompt caching is an architectural optimization in large language model inference engines that preserves pre-computed Key-Value (KV) cache tensors for repeated prefix tokens across requests.[1] By bypassing the redundant prefill computation of static context (such as system instructions, tool schemas, and base documents), prompt caching substantially reduces Time to First Token (TTFT) and token consumption costs.[2]

Prefix caching mechanics

In standard transformer inference, when a prompt arrives, the model performs matrix multiplications across all input tokens to generate intermediate Key (\(K\)) and Value (\(V\)) activation tensors for every attention head in every layer.[1]

Prompt caching identifies shared token prefixes using hierarchical tree structures (such as Radix Trees in RadixAttention):[3]

Prefix hashing: The engine hashes token ID sequences starting from index 0.
KV cache lookup: If a matching prefix tree node exists in GPU VRAM, the engine loads the pre-computed \(K\) and \(V\) tensors directly into memory without invoking model forward passes.
Suffix prefill: The engine computes attention only for the newly added dynamic tokens at the end of the prompt.[3]

Latency and economic impact

Prompt caching fundamentally shifts the economics of multi-turn conversational agents and large-document RAG:[2]

Time to First Token (TTFT): Bypassing the prefill phase for 10k to 50k tokens reduces TTFT from several seconds to tens of milliseconds.
API pricing discounts: Upstream model providers (including Anthropic, OpenAI, and Google) discount cached input tokens by 50% to 90% compared to uncached input tokens.

Structuring prompts for cache stability

Because caching relies on exact sequence prefix matching from token index 0, any mutation at the beginning of a prompt invalidates the cache for all subsequent tokens:[1]

Correct layout (cache friendly): Place static content first: [System Instructions] → [Tool Definitions] → [Few-Shot Examples] → [Document Context] → [User Input / Timestamps].
Anti-pattern (cache breaking): Injecting dynamic data (such as current timestamps, random session IDs, or volatile user state) at the top of the system prompt, which forces a complete cache miss on every request.

Cache eviction and multi-tenant security

GPU memory is finite. Runtimes manage prompt cache lifecycles using Least Recently Used (LRU) eviction policies with Time-to-Live (TTL) expiration windows.[3] In multi-tenant systems, prefix keys are partitioned by organization and workspace IDs to prevent unauthorized cross-tenant attention leakage.

References

↑ Mitra, Sampriti. System Design for the LLM Era: Patterns and Principles for Production-Grade AI Architecture. Packt Publishing, 2026. Ch. 2: "Core Architectural Patterns for LLM System Design."
↑ Anthropic. "Prompt Caching Documentation and Best Practices," 2024. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
↑ Zheng, Lianmin, et al. "SGLang: Efficient Execution of Structured Language Model Programs." arXiv preprint arXiv:2312.07104, 2023.