When building applications with Large Language Models (LLMs), you may notice users asking the same questions in slightly different ways. Each time, your application sends the query to the LLM, incurring costs and adding latency. A semantic caching LLM system addresses this by storing and reusing answers to similar, not just identical, questions.
This guide explains how semantic caching works, how to implement it safely for production, and how to avoid common pitfalls that lead to incorrect, stale, or insecure responses.
Key takeaways
- Semantic caching reuses LLM responses for similar queries to reduce latency and cost, but semantic similarity does not guarantee answer correctness.
- Safety is critical. Caching is unsafe for time-sensitive, personalized, or permission-dependent queries without strict metadata filtering and invalidation rules.
- A fixed similarity threshold is a blunt tool. Production systems require careful calibration based on the cost of a wrong answer and may use different rules for different query types.
- Effective caching is a systems problem. It requires managing embeddings, context, metadata (like user ID and model version), and the cache lifecycle (like expiration and eviction).
What Is Semantic Caching for LLMs?
Semantic caching for LLMs stores and reuses a generated response for a new request that is semantically similar, not just textually identical. This technique reduces redundant LLM API calls, which lowers both latency and operational costs. In specific question-answering workloads, this can result in up to a 50–60% reduction in redundant computation (Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models). The core trade-off is efficiency versus the risk of serving a contextually incorrect answer for a similar-sounding query.
An embedding is a numerical representation of text, capturing its semantic meaning in a vector of numbers. Semantic caching uses these embeddings to find similar past questions.
This differs from traditional, exact-match caching. An exact-match cache only works if a new query is a character-for-character match of a previous one. It misses simple paraphrases like "How do I reset my password?" if the cached query was "What's the process for a password reset?".
A semantic cache would likely identify these two questions as similar enough to reuse the answer. This is powerful but introduces a new challenge: how similar is similar enough? A small change in a query can completely change the correct answer, a problem known as a "response mismatch" (Semantic Caching for Low-Cost LLM Serving).
| Feature | Exact-Match Caching | Semantic Caching |
|---|---|---|
| Matching Logic | Requires identical query strings. | Matches based on semantic meaning (vector similarity). |
| Flexibility | Low. Misses paraphrases and minor variations. | High. Can handle paraphrases and rewordings. |
| Risk of Error | Very low. A hit is guaranteed to be for the same query. | Higher. "Similar" does not mean "equivalent." |
| Primary Use Case | Deterministic, repeatable requests. | Repetitive, informational queries where variations are common. |
How Semantic Caching Works: A Step-by-Step Guide
A semantic cache sits between your application and the LLM, intercepting requests. It can be implemented as part of an LLM gateway or as a standalone component. The process involves checking the cache for a similar query before calling the LLM.
Here is the end-to-end flow for a typical semantic caching system.
Request Normalization and Contextualization. The system first gathers all relevant context. This includes the user's prompt, recent messages in a conversation, the user's ID and permissions, the model being used (e.g., GPT-4), and its configuration. This complete package forms the basis for the cache lookup. For multi-turn conversations, the dialogue history is crucial context (Multi-Turn Semantic Caching for LLM Gateways).
Embedding Generation and Vector Search. The normalized request is converted into a vector embedding. This vector is then used to search a vector index (often using an Approximate Nearest Neighbor algorithm like HNSW) for embeddings of previously answered questions.
Similarity and Metadata Filtering. The system finds the closest match and calculates its similarity score (e.g., cosine similarity). If this score is above a pre-defined threshold, it proceeds to the next check. Critically, it must also verify that essential metadata matches exactly. For example, the user ID, tenant ID, and model version must be identical to the cached entry. A similar question from a different user must be a cache miss.
Cache Hit. If both the similarity score and metadata filters pass, the system retrieves the stored response and returns it to the user. This avoids an LLM call. The system should log the cache hit for monitoring.
Cache Miss. If no entry meets the similarity and metadata criteria, it's a cache miss. The request is sent to the LLM for a new response.
Store New Entry. Once the LLM generates a response, the system stores the request, its embedding, the response, and all relevant metadata in the cache. It also assigns a Time-to-Live (TTL), which defines how long the entry should be considered fresh.
Monitoring and Eviction. A production system continuously monitors metrics like hit rate, false-hit rate, and latency. Old or infrequently used entries are removed from the cache (evicted) to save space. Eviction policies for semantic caches should consider the cost of a mismatch, not just how recently an entry was used. Optimal offline policy selection has been described as an NP-hard problem, which is why production systems rely on polynomial-time heuristics and online policies (An Online Algorithm for Semantic Caching).
Worked Example: The Danger of a False Hit
Semantic similarity does not equal semantic equivalence. A simple example shows how relying only on a similarity score can lead to factually incorrect answers.
Imagine a semantic cache with the following entry:
- Cached Query: "What was the capital of West Germany?"
- Cached Response: "Bonn."
Now, two new queries arrive. We'll use a hypothetical cosine similarity score, where 1.0 is a perfect match.
Scenario 1: A Safe Hit
- New Query: "Which city was the capital of West Germany?"
- Similarity Score: 0.98 (very high, as it's a direct paraphrase).
- Outcome: The query is extremely similar. The cache correctly returns "Bonn." This is a true hit.
Scenario 2: An Unsafe "False Hit"
- New Query: "What is the capital of Germany?"
- Similarity Score: 0.92 (still high, due to shared words like "capital" and "Germany").
- Outcome: Let's say our similarity threshold is set to 0.90. The system sees 0.92 > 0.90 and considers it a match. It incorrectly returns "Bonn" from the cache. The correct answer is Berlin. This is a false hit.
This example shows that a similarity threshold alone is not enough. The meaning changed despite the high vector similarity. Production systems must account for this risk.
When Is Semantic Caching Unsafe?
Semantic caching is unsafe for queries that are time-sensitive, personalized, or dependent on specific user permissions. In these cases, a cached response is likely to be stale, incorrect, or a security risk. Caching should be disabled or used with extreme caution for such workloads.
Here are five scenarios where semantic caching is dangerous.
Time-Sensitive Queries
Answers about stock prices, news headlines, weather forecasts, or product inventory levels change quickly. Caching these responses, even for a few minutes, can provide users with outdated and misleading information.
Personalized or Private Data
Queries like "What is my order status?" or "Summarize my recent emails" are specific to one user. Caching the response could leak one user's private data to another who asks a similar question. Strict metadata filtering on user ID is mandatory to prevent this.
Authorization-Dependent Queries
A response can change based on a user's role. An admin asking "How many users are in the system?" should get a different answer than a guest. If the cache doesn't filter on user permissions, it could leak sensitive information to unauthorized users.
Queries with Precise Constraints
LLM embeddings can struggle with small but critical changes, especially numbers, dates, and negations. A cached answer for "Find laptops under $1000" is wrong for the query "Find laptops over $1000." Similarly, a cache can easily confuse "Which of these are included?" with "Which of these are NOT included?".
Side-Effecting Operations
If a query triggers an action, like sending an email, creating a user account, or running a database update (often via tool use), its response should not be cached. A cache hit would return the success message without performing the necessary action.
How Should You Choose a Similarity Threshold?
You should choose a similarity threshold by balancing the need to get more cache hits (recall) with the need to avoid incorrect answers (precision). There is no single "best" threshold; it depends on your specific data and the cost of a mistake. For high-stakes applications, precision is far more important than recall.
Start by creating a test set of query pairs from your application's domain. Label each pair as either "equivalent" (the same answer is correct for both) or "non-equivalent."
- Test a Range of Thresholds: For each threshold (e.g., from 0.85 to 0.99), run your test set through the cache logic.
- Measure Precision and Recall:
- True Hit: The cache correctly serves an answer for an equivalent query.
- False Hit: The cache incorrectly serves an answer for a non-equivalent query.
- Precision: Of all the hits your cache made, what percentage were correct? (True Hits / (True Hits + False Hits)).
- Recall: Of all the possible correct hits in your data, what percentage did your cache find? (True Hits / (True Hits + False Misses)).
- Prioritize Precision: In most applications, a false hit (wrong answer) is much more costly than a cache miss (an extra LLM call). Optimize for high precision first. Research on privacy-aware caching emphasizes using metrics like F-score, which combines precision and recall, rather than just the raw hit rate. For example, one system achieved approximately 17% higher F-score and 20% higher precision for cache decisions compared to the GPTCache library in its reported experiments (Privacy-Aware Semantic Cache for Large Language Models).
- Consider Different Thresholds: A one-size-fits-all threshold is often a mistake. You can use a stricter threshold (e.g., 0.98) for sensitive queries about legal or financial topics and a looser one (e.g., 0.92) for low-risk, general informational questions. Some advanced systems use more complex rules to distinguish between safe hits, new questions, and out-of-domain queries.
Common Mistakes and Limits to Know
Implementing a semantic cache seems simple, but many teams make critical errors that compromise safety and correctness. Here are common mistakes to avoid.
- Forgetting Metadata Filters: Failing to include tenant ID, user ID, or role in the cache key is a severe security flaw that can lead to data leaks.
- Ignoring Invalidation with RAG: A semantic cache reuses a prior final answer, whereas a Retrieval-Augmented Generation (RAG) system retrieves fresh source material to generate a new answer. Caching RAG responses is risky because if the underlying documents change, the cached answers become stale. You must have a strategy to invalidate cache entries when their sources are updated to ensure the cached response remains compatible with current evidence (Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models).
- Neglecting Multi-Turn Context: In a chatbot, the meaning of "What about the second one?" depends entirely on the conversation history. Using only the last message as the cache key will lead to contextually wrong answers.
- Mixing Model Configurations: Serving a response generated by
gpt-4-turbowith a specific system prompt for a request intended forclaude-3-opuswith a different prompt will produce unpredictable results. Model name, version, and system prompt must be part of the cache metadata. - Optimizing Only for Hit Rate: A high hit rate is a vanity metric if it includes many incorrect "false hits." Focus on answer correctness and task success instead.
- Using a Single Global Threshold: Applying the same similarity rule to all query types ignores their different risk profiles.
What to learn next
- LLM Gateways: Semantic caching is often a feature of an LLM gateway, which is a centralized service for managing requests to various AI models.
- Continuous Batching: To improve LLM throughput for cache misses, learn how servers can group requests together using techniques like continuous batching LLM.
- Speculative Decoding: For requests that do go to the model, techniques like Speculative Decoding: How It Works to Speed Up LLMs can significantly reduce latency.
Sources and further reading
- Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
- Multi-Turn Semantic Caching for LLM Gateways with Calibrated Out-of-Domain Rejection
- Privacy-Aware Semantic Cache for Large Language Models
- Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
- RedisVL Semantic Caching for LLMs
- An Online Algorithm for Semantic Caching
