Retrieval-Augmented Generation (RAG) systems depend on finding the right documents to answer a user's question. But what happens when the user's query is vague, complex, or uses different words than your documents? The retrieval step can fail, leading to a poor answer. RAG query rewriting is a technique that uses a language model to translate a user's question into one or more better queries, increasing the chance of successful retrieval.

Key takeaways

  • Query rewriting bridges the gap between how users ask questions and how information is stored. This helps improve the retrieval of relevant documents.
  • There is no single best rewriting technique. The right choice depends on the specific problem, whether the query is vague, complex, or uses mismatched terminology.
  • Rewriting is a tradeoff. It can improve results but adds latency and cost from extra language model calls. It also risks introducing noise or unsupported assumptions.
  • Always preserve the user's original query. Use it for generating the final answer and for auditing the system's behavior. The impact of rewriting should be measured on both retrieval metrics and end-to-end answer quality.

Why Do User Queries Fail in RAG?

A retriever fails when it cannot find relevant documents for a user's query. RAG query rewriting helps fix this by creating better search terms. A retriever can perform poorly when the user’s wording differs from the terminology used in the indexed documents (Source). This mismatch is a primary cause of many common RAG failure modes.

Here are four common reasons why an initial query might fail:

  • Vocabulary Mismatch: The user and the documents use different words for the same concept. A user might ask about a "company car," but the official policy document uses the term "fleet vehicle." A simple keyword or vector search might miss this connection.
  • Ambiguity: The query lacks necessary context, often in a conversational setting. A follow-up question like "What about the new proposal?" is impossible to answer without knowing which proposal the user means from the previous conversation.
  • Complexity: The query asks multiple distinct questions at once. For example, "What were the Q3 results and how did they compare to the Q2 forecast?" requires finding two separate pieces of information, likely in different documents.
  • Under-specification: The query is too broad and doesn't provide enough detail. A question like "Tell me about our security" forces the system to guess whether the user is interested in cybersecurity, physical security, or data privacy policies.

How Query Rewriting Works: A Step-by-Step Guide

Adding a query rewriting module to a RAG pipeline involves transforming the user's question before retrieval. This process adds a language model call but can significantly improve the quality of documents found. The goal is to provide the final generation model with more relevant context to work from.

Here is the step-by-step workflow of a RAG system with query rewriting:

  1. Analyze and Preserve the Query: The system receives the user's question. It immediately saves the original query to be used later for answer generation and logging. This ensures the final answer directly addresses what the user asked.
  2. Choose a Rewriting Strategy: The system decides how to rewrite the query. This can be a fixed strategy, like always generating multiple versions, or a dynamic one based on the query's characteristics (e.g., detecting if it's a follow-up question).
  3. Generate Rewritten Queries: A Large Language Model (LLM) transforms the original query into one or more new queries optimized for retrieval. For reliability, the LLM should be prompted to return a structured output, such as a JSON list of strings.
  4. Retrieve Documents in Parallel: The system runs a retrieval operation for each rewritten query, often including the original query. This parallel search can use vector, keyword, or hybrid search and can be combined with metadata filters to narrow results from heterogeneous collections (Source).
  5. Fuse and Rerank Results: The lists of retrieved documents from each query are combined. The system removes duplicate documents and then ranks the combined set. Techniques like Reciprocal Rank Fusion (RRF) can merge the ranked lists, and a dedicated reranker model can further refine the final evidence set by scoring document relevance against the original query.
  6. Generate the Final Answer: The top-ranked, most relevant documents are passed as context to the generator LLM. Crucially, the generator receives the original user query, not the rewritten ones. This ensures the answer is faithful to the user's initial intent.

Which Rewriting Technique Should You Use?

You should choose a rewriting technique based on the specific problem your user's query presents. Using the wrong technique can add cost and latency without improving results. For example, decomposing a simple query is inefficient, while simply expanding a complex multi-part query may not find all the necessary evidence.

The table below maps common query problems to recommended rewriting techniques and their key tradeoffs.

Query Problem Recommended Technique How It Works Key Tradeoff
Vague or conversational Standalone Rewrite An LLM uses conversation history to make the query self-contained. "What about it?" becomes "What are the security implications of the new login system?" Can invent unsupported details if conversation history is ambiguous. If antecedents are unknown, asking for clarification is safer.
Terminology mismatch Query Expansion The LLM adds synonyms or related technical terms to the original query. "Laptop issues" becomes "Laptop issues OR notebook computer troubleshooting OR hardware diagnostics." Can broaden the search too much and retrieve irrelevant documents, adding noise.
Multiple hidden questions Multi-Query Retrieval An LLM generates several semantically distinct questions to cover different facets of the original query. Techniques like DMQR-RAG focus on maximizing this diversity (Source, Source). Increases latency and cost due to parallel LLM calls and retrieval operations.
Complex multi-step question Decomposition The LLM breaks the query into a chain of simpler sub-questions. These are often executed sequentially, with the answer from one step informing the next query (Source). High latency and complex logic. An error in an early step can cause the entire chain to fail.
Exploratory or multi-step question Iterative Retrieval The system retrieves documents, uses that context to generate a new, more informed query, and repeats the process. Methods like EfficientRAG aim to make this process more efficient (Source). Higher accuracy for complex questions, but adds significant latency and complexity due to multiple sequential retrieval steps.
Poor semantic match HyDE (Hypothetical Document Embeddings) The LLM generates a fictional, "perfect" answer to the query. The system then uses the embedding of this hypothetical answer to find real documents with similar meanings (Source). The hypothetical document is not real evidence and can steer retrieval in the wrong direction if it's inaccurate or too imaginative.

A Worked Example: From Vague Query to Precise Answer

Let's walk through an example to see how RAG query rewriting works in practice. Imagine a user is in a conversation about planning for the upcoming 2026 year and has previously expressed concerns about the EU market.

1. Original Query

The user asks a vague, conversational question:

"What about our Q4 sales strategy?"

2. Problem Analysis

This query is underspecified and conversational. Without the prior context, a standard retriever would find generic documents about "sales strategy" that are not specific to the user's needs. The retrieved documents would likely be of low relevance.

3. Rewriting Step

The system identifies the query as conversational and ambiguous. It chooses a Multi-Query Retrieval strategy to cover the likely intents based on the conversation history. An LLM generates three distinct, standalone questions using the established context:

JSON
{
  "rewritten_queries": [
    "What is the sales strategy for Q4 2026?",
    "What are the risks in the EU market for the Q4 2026 sales strategy?",
    "How does the Q4 2026 sales strategy differ from the Q3 2026 strategy?"
  ]
}

4. Retrieval and Fusion

The system retrieves documents for all three queries in parallel.

  • Query 1 finds the official "Q4 2026 Sales Strategy" document.
  • Query 2 finds a "Risk Analysis Memo: EU Market Headwinds" from the relevant quarter.
  • Query 3 finds no specific comparison document.

The results are combined, and the two unique, highly relevant documents are passed to the next stage.

5. Final Answer Generation

The generator LLM receives the original query ("What about our Q4 sales strategy?") along with the content of the strategy document and the risk memo. It can now synthesize a comprehensive answer that summarizes the strategy and specifically addresses the risks in the EU market, satisfying the user's inferred intent.

What Are the Common Pitfalls of Query Rewriting?

While powerful, query rewriting can harm performance if not implemented carefully. It is not guaranteed to improve every question, and some studies report variable accuracy and context-dependent benefits (Source). Avoid these common mistakes to ensure your rewriting module helps rather than hinders.

  • Intent Drift: Do not let the LLM rewrite the query so much that it no longer reflects the user's original goal. The new query should be a clarification, not a different question.
  • Hallucinating Constraints: The LLM must not invent details. Adding a specific date, entity, or constraint that was not in the user's query or conversation history will lead to retrieving the wrong documents.
  • Discarding the Original Query: Never use a rewritten query for the final answer generation. Always generate the answer using the user's original words to remain faithful to their request.
  • Ignoring Cost and Redundancy: Every LLM call and retrieval step adds time and cost. Generating many near-duplicate queries without a diversity check adds overhead without improving coverage.
  • Over-relying on Vector Similarity: Using only vector search for queries that contain exact identifiers, numbers, code, or rare names is a frequent mistake. These often require sparse retrieval (like BM25) or hybrid search for accurate matching.
  • Evaluating Only the Final Answer: A good-sounding answer can be generated from poor evidence. It's crucial to evaluate the entire pipeline with metrics for retrieval (e.g., recall of supporting evidence, ranking quality), generation (answer correctness, citation faithfulness), and system performance (latency, token usage). See our guide to RAG evaluation metrics.
  • Assuming Generalization: Assuming a rewriting strategy that improves performance on one dataset or domain will work equally well for other corpora, languages, or question types. Rewriting performance is highly context-dependent.

What to learn next

Sources and further reading