Retrieval-Augmented Generation (RAG) systems combine the power of large language models with external knowledge bases. But how do you know if your RAG system is actually working well? The answer lies in using the right RAG evaluation metrics, which allow you to systematically test, diagnose, and improve your application.

This article explains a framework for RAG evaluation. You will learn how to assess both the retrieval and generation components of your system to understand not just if it failed, but why it failed.

Key takeaways

  • RAG evaluation is a two-part process. First, test the retriever’s ability to find relevant information. Then, test the generator’s ability to use that information correctly.
  • Core retrieval metrics are Context Precision and Context Recall. Precision asks if retrieved chunks are relevant, while Recall asks if you retrieved all the information needed.
  • Core generation metrics are Faithfulness and Answer Relevance. Faithfulness checks if the answer sticks to the provided context, and Answer Relevance checks if the answer addresses the user's question.
  • Use metrics as diagnostic tools. A low score on a specific metric points to a specific problem, such as poor document chunking, a weak embedding model, or a flawed prompt for the generator.
  • A single quality score is not enough. A full evaluation should also track performance metrics like the latency of the retrieval and generation steps.

The Two-Stage Framework for RAG Evaluation

To effectively test a RAG system, you should evaluate its two main components separately: the retriever and the generator. This separation is the key to diagnosing problems. A bad final answer could be because the retriever missed the key document, or because the language model ignored the perfectly good information it was given (Source: Microsoft Azure). Thinking in two stages tells you exactly where to look.

Stage 1: Evaluating the Retriever

The retriever is the component responsible for searching your knowledge base and finding documents, or "context," that seem relevant to the user's query. Before a single word of the answer is generated, you must assess the quality of this retrieved context.

The central question here is: Did we find the right puzzle pieces? If the retriever provides irrelevant or incomplete information, the language model has no chance of producing a correct answer, no matter how powerful it is.

Stage 2: Evaluating the Generator

Once the retriever passes over the context, the generator (a large language model) uses it to synthesize a human-readable answer. The evaluation at this stage focuses on the quality of the final output.

Here, the question is: Given the puzzle pieces it had, did the model build a good answer? This stage checks if the model's answer is faithful to the provided context, relevant to the user's question, and clearly written. A retrieval score alone doesn't guarantee a correct answer, as the generator can still omit or misinterpret the information it receives (Source: Microsoft Azure).

Python
# Conceptual two-stage evaluation
def evaluate_rag_system(query, ground_truth_docs, ground_truth_answer):
    # Stage 1: Evaluate the retriever
    retrieved_docs = retriever.find(query)
    retrieval_scores = evaluate_retriever(retrieved_docs, ground_truth_docs)

    # Stage 2: Evaluate the generator
    generated_answer = generator.answer(query, retrieved_docs)
    generation_scores = evaluate_generator(generated_answer, retrieved_docs, ground_truth_answer)

    return {**retrieval_scores, **generation_scores}

This two-stage approach allows for granular, component-level testing, which is essential for building robust RAG systems (Source: Microsoft Azure).

How Do You Build a RAG Evaluation Dataset?

The foundation of any good evaluation is a high-quality test dataset. This dataset should contain representative evaluation cases with user questions and, when possible, the expected answers and labels for which documents are relevant.

To test your system robustly, the dataset must include a diverse range of question types:

  • Normal Questions: Straightforward queries that have a clear answer in the knowledge base.
  • Ambiguous Questions: Queries that could be interpreted in multiple ways, testing the system's ability to handle nuance.
  • Multi-hop Questions: Complex questions that require synthesizing information from multiple documents to answer.
  • Unanswerable Questions: Queries whose answers are not in the knowledge base, testing the system's ability to gracefully refuse to answer rather than hallucinate.
  • Adversarial Questions: Queries designed to trick or mislead the system, testing its robustness.

How Do You Measure Retrieval Quality?

You measure retrieval quality with metrics that assess if the context given to the LLM is useful and complete. The two most common RAG evaluation metrics for this are Context Precision and Context Recall. These metrics tell you whether your retriever is finding high-quality, relevant information or just noisy, unhelpful text.

Context Precision

Context Precision measures the relevance of the retrieved documents. It answers the question: "Of all the context we retrieved, how much of it is actually useful?" A low precision score means your retriever is pulling in a lot of irrelevant text, which can confuse the LLM and lead to incorrect answers.

Think of it like a search engine. If you search for "apple pie recipes" and the first five results are about Apple computers, the precision is low.

Context Recall

Context Recall measures whether you found all the necessary information to answer the question. It asks: "Did we retrieve all the essential passages needed to fully answer the user's query?" A low recall score means your retriever is missing key facts, making a complete and accurate answer impossible.

For example, if a user asks "What were the main outcomes of the Apollo 11 mission?" and the retriever only finds documents about the launch, recall is low because information about the moon landing and return journey is missing. Context Precision focuses on the quality of what was found, while Context Recall focuses on whether everything necessary was found (Source: Ragas Docs).

How Do You Measure Generation Quality?

You measure generation quality by assessing the final answer's relationship to both the retrieved context and the user's question. Key metrics include Faithfulness, Answer Relevance, and Answer Correctness. These check if the answer is factually sound, on-topic, and ultimately right. For example, some evaluation services return bounded scores; Azure’s RAG evaluators use a 1-to-5 score range and define a default pass threshold of 3 (Source: Microsoft Azure).

Faithfulness

Faithfulness, sometimes called Groundedness, measures how factually consistent the answer is with the provided context. It asks: "Is every claim in the generated answer supported by the retrieved passages?" This is your primary metric for fighting LLM hallucinations, where a model makes up information. An answer can be well-written and sound plausible, but if it's not based on the provided sources, it fails the faithfulness test.

Faithfulness specifically checks if the answer is a logical consequence of the context, not whether it matches a pre-written "golden" answer (a single, ideal reference answer) (Source: Ragas Docs).

Answer Relevance

Answer Relevance measures if the generated response actually addresses the user's original question. It asks: "Does this answer satisfy the user's intent?" An answer can be 100% faithful to its sources but completely miss the point of the question. For example, if a user asks "What is the capital of France?" and the system faithfully uses a retrieved document to answer "France is a country in Western Europe," the answer is faithful but not relevant. This metric is distinct from faithfulness and equally important for user satisfaction (Source: Ragas Docs).

Answer Correctness

Answer Correctness measures if the information in the answer is factually accurate according to a trusted source or "ground truth" reference answer. This is different from faithfulness. An answer can be faithful to a retrieved document that is itself outdated or incorrect. In this case, the answer is faithful but not correct. For example:

  • Query: "Who is the current UK Prime Minister?"
  • Outdated Document: "Theresa May is the Prime Minister of the UK."
  • Faithful but Incorrect Answer: "According to the source, Theresa May is the current Prime Minister."

This distinction is critical. Faithfulness guards against the LLM inventing things, while correctness guards against the knowledge base itself being wrong (Source: Ragas Docs).

Reference-Based vs. Reference-Free Evaluation

Evaluation methods can be divided into two types: reference-based and reference-free.

  • Reference-based evaluation requires a "ground truth" dataset containing ideal answers or labeled documents. Metrics like Answer Correctness and traditional retrieval scores (e.g., precision@k) rely on these references for direct comparison. This method is accurate but can be expensive, as creating high-quality reference data is time-consuming.
  • Reference-free evaluation does not require a pre-written answer. Instead, it often uses a powerful LLM (an "LLM-as-a-judge") to score the quality of an answer based on the query and retrieved context alone. Metrics like Faithfulness and Answer Relevance are often calculated this way. This approach is more scalable but depends heavily on the quality and impartiality of the judge model (Source: Hugging Face).

A balanced strategy uses both methods: reference-based evaluation for core accuracy checks and reference-free methods to assess quality at scale.

A Diagnostic Guide to RAG Evaluation Metrics

Low scores on specific RAG evaluation metrics are not just grades; they are symptoms of underlying problems. Use this table as a first step to diagnose issues in your RAG pipeline.

Symptom (Low Score) Potential Problem Area Where to Look First
Low Context Recall Retrieval (Missing Info) Document Chunking: Are your chunks too small, splitting related facts? Try larger chunks or sentence-aware chunking.
Embedding Model: Does your model struggle with your domain's specific terminology (e.g., medical or financial terms)? Consider a domain-specific model or fine-tuning vs RAG.
Search Index: Is your document processing pipeline (e.g., cleaning HTML) dropping key information?
Low Context Precision Retrieval (Noisy Info) Top-K Setting: Are you retrieving too many documents (e.g., k=10)? Try reducing k to 3 or 5.
Reranking: Add a reranker model after the initial retrieval to push more relevant results to the top.
Low Faithfulness Generation (Hallucination) Prompt Engineering: Is your prompt too "creative"? Instruct the model explicitly to only use the provided sources.
Model Choice: Some models are more prone to hallucination. Test a different generator model.
Low Answer Relevance Generation (Off-Topic) Prompt Engineering: Does your prompt clearly state the user's original question? Re-emphasize the core task.

Beyond Averages: Reporting and Regression Testing

A robust evaluation strategy goes beyond calculating a single average score for your test set.

Aggregate and Slice Results: Instead of relying on one mean score, aggregate results across your entire test set but also inspect failures on individual questions. Analyze performance on different "slices" of your data, such as by question type (e.g., multi-hop vs. simple), document source, or topic. Report distributions and confidence intervals, not just averages, to understand the variance in your system's performance (Source: Microsoft Azure).

Implement Regression Testing: Integrate RAG evaluation into your development workflow. Create a fixed, representative test set (a "golden set") and run it automatically whenever a change is made to your system—whether to the chunking strategy, embedding model, or prompt. By setting acceptance thresholds for your key metrics, you can automatically catch "regressions" where a change unexpectedly hurts performance, creating a CI/CD-like process for quality control (Source: DeepEval).

Common Mistakes in RAG Evaluation

When starting with RAG testing, it's easy to fall into common traps. Avoiding these will make your evaluation process much more reliable.

  1. Evaluating only the final answer. The biggest mistake is to ignore the retrieved context. If you only look at the final answer, you will never know if a failure was caused by bad retrieval or bad generation. Always save and inspect the context.
  2. Confusing faithfulness with correctness. As we've seen, an answer can be 100% faithful to a source that is wrong. Never assume a "faithful" answer is a "correct" one. Always validate against ground truth when accuracy is critical.
  3. Creating reference answers with the same model you are testing. Using the same LLM to generate "golden" answers and to score your system's output can create a bias, as the model is essentially grading its own work based on its own style and assumptions.
  4. Not testing for "no answer." A good RAG system should know when it doesn't know. Your test set must include unanswerable questions to see if your system correctly refuses to answer instead of hallucinating.
  5. Ignoring latency. A high-quality answer that takes 30 seconds to generate may be unusable in a real-world application. Always track the performance (latency) of both the retriever and the generator alongside quality metrics.

What to learn next

Sources and further reading