Prompt caching is a technique used by Large Language Models (LLMs) to speed up responses and reduce costs. It works by saving the intermediate computations for the beginning of a prompt and reusing them in subsequent requests. When an application sends many prompts that start the same way, this prompt caching LLM feature avoids redundant processing.
This guide explains exactly how it works, how to use it effectively, and what common mistakes to avoid.
Key takeaways
- Prompt caching reuses the computed state of a prompt's beginning (the prefix), not the model's final answer. This reduces cost and latency on subsequent, similar requests.
- It requires an exact token-for-token match to work. To maximize cache hits, you must place stable content like system instructions first and dynamic user input last.
- The benefits and rules vary significantly between providers. Details like minimum token length, cache lifetime, and pricing are not universal.
- This technique is different from other optimizations. It is not response caching (reusing old answers) or the standard KV cache LLMs use within a single generation.
- Incorrect implementation can lead to low hit rates and negligible savings. It can also create security risks if sensitive data is cached without proper user isolation.
What Is Prompt Caching? (And What It Isn't)
Prompt caching, also called prefix caching, reuses the intermediate computational state for a recurring sequence of tokens at the start of a prompt. The system stores this state after the first request and, for subsequent requests with the same prefix, loads the state instead of recomputing it. This saves processing time and can lower input token costs, but the model still generates a new response based on the full prompt (Source name).
A common misconception is that prompt caching stores and returns a previously generated answer. It does not. It only recycles the prompt processing work, not the response generation work (Source name).
The table below clarifies the differences between various caching types.
| Caching Method | What It Stores | When It's Used | Primary Benefit |
|---|---|---|---|
| Prompt (Prefix) Caching | The intermediate KV state of a prompt's initial token sequence (prefix). | Across different API requests that share an identical prefix. | Reduces prompt processing (prefill) latency and input token cost. |
| Response Caching | The final, complete LLM-generated response. | For identical requests where a stored answer is acceptable. | Eliminates all LLM processing for repeat questions. |
| Within-Request KV Caching | The intermediate KV state of tokens within a single generation. | During the generation of one response, to avoid recomputing early tokens when predicting later ones. | Speeds up the generation of long sequences in a single request. |
How Does Prompt Caching Work? A Step-by-Step Guide
Prompt caching works by saving the key-value (KV) tensors from the Transformer model's attention mechanism for a prompt's prefix. This saves the computationally intensive "prefill" step for that prefix on future calls. When a new request arrives with an identical starting sequence of tokens, the system loads the saved tensors and begins computation only on the new part of the prompt (Source name).
Here is the process step-by-step:
- Initial Request Processing: Your application sends the first request with a long prompt. The LLM inference engine tokenizes the text and performs the "prefill" computation. During this phase, it processes all prompt tokens simultaneously to generate a key-value (KV) state for each one. This state captures the relationships between tokens in the prompt.
- Cache Write: The inference engine identifies a cacheable prefix from the prompt. On this first request, it computes the KV state for this prefix and stores it in a dedicated cache. Some providers may charge a different, higher rate for this initial "cache write" (Source name).
- Subsequent Request Arrives: Your application sends a second request. The prompt for this request begins with the exact same text as the first one but ends with a different user question.
- Cache Hit: The inference engine tokenizes the new prompt and compares its initial token sequence against the stored prefixes in the cache. It finds an exact match. This is a
cache hit prompt. - State Loading and Suffix Computation: Instead of recomputing the KV state for the entire matching prefix, the system loads the stored state directly from the cache. It then performs the prefill computation only for the new part of the prompt (the suffix). From there, it proceeds to the generation phase to create the response.
This process skips the most computationally expensive part of handling a long prompt, directly reducing the time it takes to generate the first token of the response.
Request 1: [Stable Prefix (2500 tokens)] + [User Question A (50 tokens)]
|---------------------------| |---------------------------|
`-- Prefill computed & KV State stored in cache (Cache Write)
Request 2: [Stable Prefix (2500 tokens)] + [User Question B (60 tokens)]
|---------------------------| |---------------------------|
`-- KV State loaded from cache `-- Prefill computed only for
(Cache Hit / Cache Read) these 60 new tokensHow Much Can Prompt Caching Reduce Cost and Latency? (A Worked Example)
Prompt caching can significantly lower both cost and latency, but the exact savings depend on your prompt structure, request volume, and the provider's pricing model. The main latency benefit comes from a lower Time To First Token (TTFT), as the model skips the prefill work for the cached part of the prompt (Source name). The cost savings come from special pricing for cached tokens.
Let's walk through an example for a simple Retrieval-Augmented Generation (RAG) application.
Scenario:
- Application: A Q&A chatbot that answers questions about a specific document.
- Stable Prefix: A 2,500-token section containing system instructions and the document's text.
- Dynamic Suffix: A 50-token user question that changes with every request.
- Volume: 100 API calls.
- Illustrative Pricing:
- Standard Input Tokens: $0.50 per 1 million tokens (Mtok)
- Cache Write Tokens: 1.25x standard price ($0.625 / Mtok)
- Cache Read Tokens: 0.1x standard price ($0.05 / Mtok)
Calculation Without Prompt Caching
Each of the 100 requests processes the full prompt from scratch.
- Total tokens per request: 2,500 (prefix) + 50 (suffix) = 2,550 tokens
- Total input tokens for 100 requests: 100 * 2,550 = 255,000 tokens
- Total Cost: (255,000 / 1,000,000) * $0.50 = $0.1275
Calculation With Prompt Caching
The first request pays a premium to write the prefix to the cache. The next 99 requests get a discount by reading the prefix from the cache. All 100 requests pay the standard rate for the unique suffix.
Cost of First Request (Cache Write):
- Prefix cost: (2,500 / 1,000,000) * $0.625 = $0.0015625
- Suffix cost: (50 / 1,000,000) * $0.50 = $0.000025
- Total for 1st request: $0.0015875
Cost of Next 99 Requests (Cache Reads):
- Prefix cost per request: (2,500 / 1,000,000) * $0.05 = $0.000125
- Suffix cost per request: (50 / 1,000,000) * $0.50 = $0.000025
- Total per cache-read request: $0.00015
- Total for 99 requests: 99 * $0.00015 = $0.01485
Total Cost: $0.0015875 (first request) + $0.01485 (next 99) = $0.0164375
Result: In this scenario, using prompt caching reduces the input token cost from $0.1275 to $0.0164, an 87% savings. The latency for the 99 cache-hit requests would also be lower due to skipping the prefill step for 2,500 tokens. For details on calculating these costs, see the guide on LLM cost per token.
How Should You Structure Prompts for a High Cache-Hit Rate?
To achieve a high cache-hit rate, you must put static, unchanging content at the beginning of your prompt and place dynamic, variable content at the end. A cache hit requires the start of your prompt to be an identical, token-for-token match with a previously cached prefix. Even a single changed character or whitespace can break the match (Source name).
Follow this hierarchical structure for your prompts:
- Stable System Instructions: The core identity, rules, and persona of your AI. This rarely changes.
- Stable Tool/Function Definitions: If using function calling, place the definitions here. These should be consistent across calls.
- Stable Few-Shot Examples: Provide consistent examples to guide the model's output format or tone. The guide on What Is Prompt Engineering? A Practical Introduction covers this topic in more detail.
- Stable Context/Documents: Include any reference text, like a document in a RAG system, that will be reused across multiple related questions.
- Volatile/Dynamic User Input: Place the user's question, conversation history, or any other data that changes with each request at the very end.
Here is a simple Python example of assembling a prompt for caching:
# This part is stable and can be cached
STABLE_PREFIX = """
System: You are a helpful assistant that answers questions about the provided document.
Do not use outside knowledge.
Document:
The Magna Carta was issued in June 1215 and was the first document to put into
writing the principle that the king and his government was not above the law.
It sought to prevent the king from exploiting his power, and placed limits
of royal authority by establishing law as a power in itself.
---
"""
# This part is dynamic and changes with each request
dynamic_user_question = "User: When was the Magna Carta issued?"
# The final prompt places the dynamic part at the end
final_prompt = STABLE_PREFIX + dynamic_user_questionTroubleshooting Low Cache-Hit Rates
If you are not seeing the expected performance or cost benefits, check your prefix for hidden dynamic content:
- Timestamps or Request IDs: Never place these at the start of a prompt.
- User or Session IDs: These should only be included at the end if necessary.
- Inconsistent Whitespace: Ensure formatting is identical every time.
- Changing JSON Key Order: When providing structured data, serialize dictionaries with a consistent key order.
- Tokenization Differences: Minor changes in text can lead to different tokens. Use a consistent method to construct your prompt string. The guide to LLM tokenization provides more information on this process.
A Comparison of Prompt Caching Across Major Providers
The rules and features for prompt caching are not standardized. They vary by LLM provider, model, and even API version. The table below summarizes documented features from several major providers to highlight these differences. Always consult the official documentation for the most current details.
| Feature | OpenAI (GPT-5.6 Doc) | Anthropic | Azure OpenAI | Self-Hosted (vLLM) |
|---|---|---|---|---|
| Minimum Tokens | 1,024 tokens (Source) | Not specified; applies to a prefix of tools, system prompt, or messages. | 1,024 tokens for supported configs (Source) | Not specified; automatic detection. |
| Matching Granularity | Not specified. | Not specified. | 128-token increments after the initial 1,024 (Source) | Exact token-prefix match. |
| Cache Control | Implicit (automatic). | Explicit (cache_control parameter) (Source). |
Implicit (automatic). | Implicit (enabled via engine config) (Source). |
| Cache Lifetime (TTL) | 30 minutes default (Source). | User-specified (e.g., 5 min, 1 hour) (Source). | Not specified. | Tied to the vLLM engine's lifetime. |
| Pricing Model | Discount up to 95% on cached tokens (Source). | Separate pricing for cache writes (e.g., 1.25x) and cache reads (e.g., 0.1x) (Source). | Not specified. | Not specified. |
What Are the Common Mistakes and Limits of Prompt Caching?
While powerful, prompt caching has limitations and is easy to misuse. Avoiding these common mistakes is crucial for getting any benefit.
Expecting Semantic Matches: A frequent error is assuming that prompts with similar meanings will share a cache. Prompt caching relies on an exact token prefix match. Paraphrasing a sentence, changing punctuation, or altering whitespace will result in a cache miss (Source name).
Placing Dynamic Data First: As highlighted earlier, inserting variable information like a timestamp or user ID at the start of the prompt is the most common way to prevent caching entirely. This breaks the prefix match immediately.
Expecting a Hit on the First Request: The first request with a new prefix cannot be a cache hit. It is responsible for warming the cache by computing the prefix's state and writing it for later use. Only subsequent requests with the same prefix can achieve a hit.
Ignoring Minimum Thresholds: Some providers only enable caching for prompts longer than a certain threshold, such as 1,024 tokens (Source name). Attempting to cache very short prefixes may not work or provide any benefit.
Forgetting Cache Eviction: Cache entries are not permanent. They have a limited lifetime (Time To Live, or TTL) and can be evicted based on provider policies. If your application sends matching requests too infrequently, the cache entry may have expired, forcing a re-computation.
Assuming It's Always Cheaper: Caching is not free. Providers may charge for cache writes. If a prefix is written but never read again, you may end up paying more than you would have without caching. Always analyze your traffic patterns to ensure prefixes are reused enough to justify the initial write cost (Source name).
Failing to Monitor: Most providers include metadata in their API responses indicating whether a cache hit occurred and how many tokens were read from the cache. If you don't monitor this information, you have no way of knowing if your caching strategy is actually working.
Security and Privacy Risks of Caching LLM Prompts
Prompt caching introduces new security and privacy considerations that developers must address. Because it involves storing parts of prompts in memory for reuse, it can create risks if not implemented carefully.
- Cross-Tenant Data Leakage: In a multi-tenant application, there is a risk that one user's request could access a cache entry created by another. This could expose sensitive data. It is critical to verify how your LLM provider isolates cache entries—whether by API key, user ID, or another method.
- Sensitive Data in Prefixes: If a stable prefix contains personally identifiable information (PII) or confidential business data, caching extends the lifetime of that data in the provider's memory. Assess whether the data in your prefix is suitable for caching under your data governance policies.
- Cache Invalidation Failures: If the source data for a cached prefix changes (e.g., a document is updated), you must have a strategy to invalidate the old cache entry. Failing to do so can cause the LLM to generate responses based on stale, incorrect information.
What to learn next
- The KV Cache LLM Explained: A Complete Technical Guide: Understand the low-level caching mechanism that makes both prompt caching and single-request generation efficient.
- LLM Cost Per Token: A Practical Calculation Guide: Dive deeper into how LLM API costs are calculated and how to model the financial impact of optimizations like caching.
- LLM Tokenization Explained: A Complete Guide: Learn why "exact text match" translates to "exact token match" and how subtle text changes can break your cache.
