Retrieval-Augmented Generation (RAG) systems give large language models (LLMs) access to external knowledge. The quality of the final answer depends heavily on the quality of the retrieved information. While a fast initial search is good at finding a broad set of potentially relevant documents, it often fails to place the most relevant document at the very top. This is where RAG reranking adds a crucial second layer of quality control.
Key takeaways
- Reranking is a second stage in a RAG pipeline that reorders a set of candidate documents fetched by a fast initial retriever.
- The most common type of reranker is a cross-encoder, which processes the query and each document jointly to produce a highly accurate relevance score.
- Reranking improves precision by promoting the best documents to the top. It cannot, however, find documents that the initial retriever failed to find in the first place.
- The primary trade-off is between better relevance and increased latency and computational cost. Applying a powerful reranker to many documents can slow down your system.
- Tuning a reranker involves finding the right balance between the size of the initial candidate pool and the final number of documents sent to the LLM.
What is RAG Reranking? A Two-Stage Process
RAG reranking is a second filtering stage that re-scores and reorders an initial list of candidate documents to improve relevance. It uses a more computationally expensive but powerful model to examine a small batch of promising documents, ensuring the best possible evidence is passed to the language model for answer generation. This creates a two-stage retrieval process.
Most RAG systems begin with a fast, efficient retriever like a vector database or a keyword search system (like BM25). This first stage is optimized for recall—its job is to quickly find a large set of documents that might contain the answer, minimizing the chance of missing the correct one.
However, this speed comes at the cost of precision. The top-ranked document from this first pass may only be superficially related to the query. Applying a powerful, slow model to every document in your entire knowledge base would be too expensive and time-consuming (Source name).
The two-stage solution is:
- Stage 1: Retrieval. A fast retriever casts a wide net, fetching an initial set of candidates (for example, one study used 50).
- Stage 2: Reranking. A slower, more accurate model carefully examines only those candidates and reorders them based on a more nuanced understanding of relevance. A smaller final set is then sent to the LLM.
This approach balances speed and accuracy, applying the most powerful analysis only where it's most needed.
How Does a Reranker Work, Step-by-Step?
A reranker re-evaluates an initial set of documents using a more sophisticated model, typically a cross-encoder, to generate new relevance scores. It then sorts the documents by these new scores to produce a more precise final ranking. This process allows for a deeper semantic analysis than the initial, faster retrieval method.
The most common reranking models are called cross-encoders. To understand how they work, it helps to first understand the embedding models, or bi-encoders, used in initial retrieval.
A bi-encoder (like those in many vector search systems) creates a numerical vector representation (an embedding) for the query and for each document independently. The system then finds documents whose vectors are mathematically closest to the query vector. This is very fast and scalable.
A cross-encoder works differently. It takes the query and a single document together as a combined input. This allows the model to look at the interaction between the words in the query and the words in the document, leading to a much more accurate relevance judgment (Source name).
Here is the typical step-by-step workflow:
- Initial Retrieval: A user submits a query. A fast retrieval system (e.g., vector search) finds the top
Ncandidate documents. Let's sayN=50. - Cross-Encoder Scoring: The reranker receives the query and the 50 candidate documents. It processes each
(query, document)pair one by one through the cross-encoder model. Each pair gets a single output score, which represents how relevant that document is to that specific query. - Reordering: The 50 documents are sorted in descending order based on their new cross-encoder scores. The original order and scores from the retrieval stage are discarded.
- Selection: The system selects the new top
kdocuments from the reranked list, where the finalkis commonly smaller than the initialN(Source name). This final, highly-relevant set of documents becomes the context for the LLM.
A Worked Example: Before and After Reranking
A concrete example shows how RAG reranking can correct the initial retrieval order and provide a better context to the LLM. Imagine a developer is asking a RAG system about a popular programming library.
User Query: "What component manages state in a React functional component?"
Step 1: Initial Retrieval (Before Reranking)
A vector search system retrieves four document chunks. It ranks them based on cosine similarity, a common metric for comparing vector embeddings. The results are good, but the best answer is buried at rank #3.
| Rank | Document Chunk | Similarity Score (Illustrative) |
|---|---|---|
| 1 | "State management is a key concept in React. Components can have their own state." | 0.89 |
| 2 | "Functional components were introduced in React 16.8 with the addition of Hooks..." | 0.85 |
| 3 | "The useState Hook is a special function that lets you add React state to functional components." |
0.82 |
| 4 | "In class components, you manage state with this.state and this.setState." |
0.78 |
Without reranking, the LLM would receive these documents in this order. The context starts with general information and only gets to the specific answer (useState) in the third document.
Step 2: Reranking and Selection
A cross-encoder reranker now processes the query with each of the four chunks. It produces a new set of more accurate relevance scores.
| Original Rank | Document Chunk | Reranker Score (Illustrative) | New Rank |
|---|---|---|---|
| 3 | "The useState Hook is a special function that lets you add React state to functional components." |
0.98 | 1 |
| 2 | "Functional components were introduced in React 16.8 with the addition of Hooks..." | 0.91 | 2 |
| 1 | "State management is a key concept in React. Components can have their own state." | 0.75 | 3 |
| 4 | "In class components, you manage state with this.state and this.setState." |
0.12 | 4 |
The reranker correctly identifies that Document #3 is the most direct and precise answer to the query. It also correctly down-ranks the document about class components as less relevant.
Final Context for LLM (After Reranking): If the system is configured to send the top two documents, the LLM now receives a much cleaner context:
- "The
useStateHook is a special function that lets you add React state to functional components." - "Functional components were introduced in React 16.8 with the addition of Hooks..."
This focused context makes it significantly easier for the LLM to generate a correct and concise answer.
How Much Does a Reranker Improve RAG Answers?
A reranker can significantly improve retrieval metrics, which often leads to better final answers, but the exact gains depend heavily on the specific system and data. Published research shows notable increases in retrieval precision and modest gains in end-to-end answer quality, accompanied by an increase in latency.
In one study, adding a reranker on top of a hybrid retrieval system increased Recall@5 (the chance that a correct document is in the top 5) from 0.695 to 0.816. The same experiment saw MRR@3 (a metric rewarding high ranks for the first relevant document) improve from 0.433 to 0.605 (Source name). These are substantial improvements in retrieval quality.
However, better retrieval doesn't always translate into proportionally better answers. In a separate controlled experiment using the Natural Questions dataset, adding a BERT-based cross-encoder reranker increased the Exact Match score of the final answer from 0.075 to 0.095. The token F1 score, which measures word overlap, rose from 0.029 to 0.040 (Source name). While these are positive gains, they are more modest than the retrieval improvements. Final answer quality is affected by many factors beyond retrieval, such as the LLM's ability to synthesize information and the structure of the prompt (Source name).
These benefits come with a cost. The same Natural Questions report measured an average of 4.3 seconds of added latency per query when running the reranker on a CPU. This dropped to below 200 milliseconds on a GPU, highlighting the importance of hardware for performance-sensitive applications (Source name). The cost of inference grows with the number of documents you rerank (Source name).
Choosing a Reranker: Cross-Encoder vs. LLM
When adding RAG reranking, you must choose a model to perform the scoring. The most common choice is a specialized cross-encoder, but using a general-purpose LLM is another popular option. Each has distinct trade-offs in accuracy, speed, and cost.
Key Considerations
Specialized Cross-Encoders like those based on BERT are trained specifically for the task of judging sentence-pair relevance. They are the standard choice for many production systems because they offer a good balance of high accuracy and manageable performance, especially with appropriate hardware. As described in the research, they work by jointly processing the query and a candidate document, which generally provides more accurate relevance judgments than the bi-encoder models used for initial retrieval. This accuracy comes at a higher computational cost per document, which is why they are used on a smaller number of candidates (Source name).
LLMs as Rerankers use a powerful generalist model to score documents, often by asking it to rate relevance on a scale or output a JSON object. This can be a flexible approach, but its performance, latency, and cost trade-offs are not covered by the cited research and must be evaluated independently. These models may also exhibit different behaviors, such as sensitivity to the order in which documents are presented.
For most teams starting out, a specialized cross-encoder provides a practical blend of performance and accuracy that is well-documented in existing research.
What Are the Limits and Common Mistakes?
Reranking is a powerful technique, but it is not a silver bullet. Misunderstanding its role can lead to wasted effort and poor results. Avoiding common mistakes is key to building a robust RAG system.
Here are some frequent pitfalls:
Mistake: Assuming the reranker can find missing documents. A reranker can only reorder the documents it is given. If the initial retrieval stage fails to fetch the correct document into the candidate pool, the reranker has no way to find it (Source name). Always evaluate your first-stage recall first. If the answer isn't in the initial candidate set, you need to improve your retriever, not your reranker.
Mistake: Ignoring latency and cost. Scoring documents with a cross-encoder is computationally intensive. Reranking 100 documents will take roughly ten times longer than reranking 10. Failing to measure and budget for this added latency can make your application feel unresponsive. Start with a small candidate pool and increase it only if you see a clear benefit.
Mistake: Treating high scores as absolute truth. A high reranker score indicates strong topical relevance between the query and the document. It does not guarantee the document is factually correct, complete, or up-to-date. The information could be subtly wrong or missing crucial context.
Mistake: Not tuning the candidate pool size. There is no universal "best" number of documents to retrieve and rerank. Retrieving too few (
N=5) may not provide enough material for the reranker to make a meaningful difference. Retrieving too many (N=200) can needlessly increase latency. The optimal numbers depend on your data and performance goals and should be found through systematic RAG evaluation.Mistake: Neglecting document quality. Reranking operates on the document chunks you provide. If your RAG chunking strategy creates messy, incomplete, or duplicative passages, the reranker will score those messy passages. Garbage in, garbage out still applies.
What to learn next
- RAG Query Rewriting: A Practical Guide - Learn how RAG query rewriting fixes retrieval problems. This guide covers key techniques, practical examples, and a framework for choosing t
- Agentic RAG: A Practical Guide to How It Works - Learn how Agentic RAG works. This guide explains the iterative retrieval loop, compares it to traditional RAG, and covers key trade-offs lik
- RAG Evaluation Metrics: A Practical Guide: Learn how to measure the performance of your RAG pipeline, including specific metrics for retrieval and generation stages.
- RAG Chunking Strategies: A Practical Guide: Discover how splitting your documents into the right-sized pieces is fundamental to retrieval quality.
- Vector Database Explained: How Similarity Search Works: Understand the technology that powers the fast first-stage retrieval in most modern RAG systems.
Sources and further reading
- From Traditional Retrieval Augmented Generation to Agentic RAG
- Evaluating the Impact of Cross-Encoder Re-Ranking on Retrieval-Augmented Generation Performance
- From BM25 to Corrective RAG: Benchmarking Retrieval Strategies
- Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking
- Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
