Standard Retrieval-Augmented Generation (RAG) systems find relevant text passages using vector search. This works well for direct questions but can fail when an answer requires connecting multiple facts scattered across different documents. This is the problem Graph RAG was designed to solve. This article offers a clear graph RAG explained, detailing how it works, its costs, and when its complexity is justified.

Key takeaways

  • Graph RAG builds a knowledge graph. Instead of just storing text, it identifies entities (people, places, concepts) and their relationships from your documents to create a structured map of your data.
  • It excels at complex questions. It is designed for questions that require connecting multiple pieces of information, a process known as multi-hop retrieval, or understanding themes across an entire document set (Source).
  • It is not a direct replacement for vector search. For simple fact lookups, traditional vector RAG is often faster and cheaper. The best approach depends on the types of questions you need to answer.
  • The main trade-off is high upfront cost. Extracting entities, building the graph, and creating summaries is computationally expensive compared to the simpler process of embedding text chunks for vector search.

How Does Graph RAG Work? From Documents to Answers

Graph RAG works by transforming your unstructured documents into a structured knowledge graph during an offline indexing phase. A knowledge graph is a network of entities (nodes) connected by their relationships (edges). At query time, the system uses this graph to retrieve highly relevant, connected information instead of just text chunks that are semantically similar to the query.

The process generally happens in two main stages: offline indexing and online querying.

Indexing Phase (Offline)

This is the computationally intensive part where the knowledge graph is built.

  1. Document Parsing: Documents are broken down into smaller text units, similar to the process in other RAG chunking strategies. Crucially, the system maintains a link from all future extracted information back to its original source passage for verification (Source).
  2. Entity & Relationship Extraction: An information-extraction process, often involving a Large Language Model (LLM), reads each text chunk and extracts key entities (e.g., "Project Alpha," "Alice," "Python") and the relationships between them (e.g., "Alice" LEADS "Team Thor," "Team Thor" MANAGES "Project Alpha").
  3. Graph Construction: The extracted entities and relationships are assembled into a a knowledge graph. This process includes entity resolution, where variations like "Microsoft" and "MSFT" are merged into a single node.
  4. Community Detection: The system identifies clusters of densely connected nodes in the graph, known as communities. These communities represent broader topics or themes. In the Microsoft GraphRAG implementation, these are organized hierarchically, from small, specific clusters to larger, more abstract ones (Source).
  5. Community Summarization: The system generates a summary for each detected community. This summary acts as a pre-computed report on a topic, allowing the system to answer broad questions without needing to read every related text chunk at query time.

Query Phase (Online)

When a user asks a question, the system retrieves context from the pre-built graph.

  1. Retrieval Strategy: Based on the user's question, the system chooses a retrieval strategy. It might perform a "local" search for specific entities and their direct connections or a "global" search that uses the pre-computed community summaries.
  2. Context Assembly: The retrieved information—which could be a subgraph, community summaries, and the original text passages—is gathered. Linking back to the source text ensures that all information remains verifiable, a key feature known as provenance (Source).
  3. Answer Synthesis: The assembled context is fed into an LLM, which synthesizes a final, human-readable answer grounded in the retrieved data.

Worked Example: Vector RAG vs. Graph RAG

To understand the practical difference, imagine a small corpus of three documents about project management.

Corpus:

  • Doc 1: "Project Alpha uses the Python programming language."
  • Doc 2: "The project is managed by Team Thor."
  • Doc 3: "Alice is the lead for Team Thor."

Let's see how Vector RAG and Graph RAG handle three different types of questions.

Question 1: Simple Fact Lookup

Query: "What language does Project Alpha use?"

  • Vector RAG: A vector search on the query will easily find Doc 1 as the most similar chunk. The LLM receives "Project Alpha uses the Python programming language" and correctly answers "Python."
  • Graph RAG: The system identifies the "Project Alpha" entity in the query. It retrieves the node for "Project Alpha" and its direct connections, finding the relationship (Project Alpha) -[USES_LANGUAGE]-> (Python). It then generates the same correct answer.

Winner: Both are effective. Vector RAG is likely faster and cheaper for this simple case.

Question 2: Multi-Hop Question

Query: "Who leads the team managing Project Alpha?"

  • Vector RAG: This is harder. The system might retrieve Doc 2 ("...managed by Team Thor") and Doc 3 ("Alice is the lead for Team Thor"). However, it has no explicit instruction to connect "Team Thor" from both documents. The LLM might fail to make the connection or hallucinate an incorrect one.
  • Graph RAG: This is where Graph RAG shines. It finds the "Project Alpha" node and traverses the graph:
    1. Project Alpha is MANAGED_BY Team Thor (from Doc 2).
    2. Team Thor is LED_BY Alice (from Doc 3). By following this two-step path (a multi-hop), the system confidently retrieves the connected facts and provides them to the LLM to correctly answer "Alice." The graph makes the relationships explicit (Source).

Winner: Graph RAG.

Question 3: Global Corpus Analysis

Query: "Give me an overview of the projects and leadership."

  • Vector RAG: A vector search might retrieve all three documents, but as disconnected snippets. The LLM would have to piece together the narrative from scratch, potentially missing the overall structure.
  • Graph RAG: The system can use its pre-built community summaries. It might have identified a "Project Alpha Management" community containing all three entities and their relationships. By retrieving the summary for this community, it can provide a coherent overview: "Project Alpha is a Python-based project managed by Team Thor, which is led by Alice."

Winner: Graph RAG. This was a key motivation for its development—to answer broad questions that resemble summarization rather than simple fact retrieval (Source).

You should choose Graph RAG when your questions require connecting disparate information or understanding the big picture, and you can justify the high upfront indexing cost. Vector RAG remains a strong choice for simpler, direct fact retrieval where speed and cost are primary concerns. Many systems use a mix, for example using a hybrid search approach.

The choice depends entirely on your data and the questions you expect users to ask.

Query Type When to Use Vector RAG When to Use Graph RAG
Simple Fact Retrieval Best fit. Ideal for questions where the answer exists in a single, self-contained text chunk. It's fast, cheap, and effective. Overkill. Can answer the question but adds unnecessary complexity and cost. The graph structure provides no extra benefit here.
Multi-Hop Questions Can fail. May retrieve relevant chunks but struggles to connect the information between them, leading to incomplete or wrong answers. Best fit. Explicitly designed to traverse relationships. It follows the chain of connections to find answers that require reasoning across documents.
Global Corpus Analysis Limited. Retrieves a list of relevant chunks, but lacks the high-level structure to synthesize a coherent summary of the entire corpus. Best fit. Uses pre-computed community summaries to provide holistic overviews and answer broad, thematic questions about the entire dataset (Source).

What Are 'Local' vs. 'Global' Search in Graph RAG?

The distinction between local and global search is a key feature of Microsoft's GraphRAG implementation that allows it to efficiently answer both specific and broad questions. These two modes operate on different parts of the indexed data structure to provide context to the LLM (Source).

Local search provides context for focused questions about specific entities. When a query mentions an entity, this mode retrieves the subgraph immediately surrounding that entity's node. It gathers the entity itself, its direct connections, and the entities they connect to. This creates a highly relevant, localized "neighborhood" of facts. For example, for the query "What is Project Alpha related to?", a local search would retrieve the Project Alpha node and its connections, like USES_LANGUAGE and MANAGED_BY.

Global search addresses broad, sensemaking questions about the entire dataset, like "What are the main security risks mentioned in these reports?" Instead of retrieving individual text chunks, it operates on the hierarchical community summaries created during indexing. The system can start at the highest-level summary and dynamically traverse down into more specific child communities that are relevant to the query, pruning irrelevant branches (Source). For very broad questions, it can use a map-reduce process: generate partial answers from multiple community reports and then combine them into a single, comprehensive final response.

What Are the Common Mistakes and Limits of Graph RAG?

Graph RAG is powerful but not foolproof. The quality of its answers is entirely dependent on the quality of the underlying knowledge graph. If the graph is flawed, the answers will be too. Here are some common failure modes and limitations to watch for.

  • Extraction Errors: The graph is only as good as the ability of the extraction process to identify entities and relationships correctly. If the model misidentifies an entity or misunderstands a relationship, that error becomes a permanent, misleading part of your graph (Source).
  • Graph Incompleteness: A failed query doesn't mean the answer isn't in your documents. It might just mean a crucial relationship or entity was missed during extraction. Real-world knowledge graphs are often sparse, meaning many true relationships are not captured, which breaks multi-hop reasoning chains (Source).
  • Poor Entity Resolution: Failing to merge duplicate entities (e.g., 'IBM', 'International Business Machines', and 'I.B.M.') fragments the graph. A query about 'IBM' might miss crucial information connected only to 'International Business Machines'.
  • Assuming a Single Architecture: "Graph RAG" is not one standardized algorithm but a family of approaches. Different systems vary in graph construction, traversal, summarization, and other features (Source).
  • Inadequate Evaluation: Focusing only on the fluency of the final answer is a common mistake. Robust evaluation must also measure factual accuracy, evidence coverage, citation correctness, retrieval recall, latency, and cost.
  • Over-reliance on Summaries: Using global summaries for precise questions can lead to factual inaccuracies or loss of detail. Summaries are abstractions. For high-precision tasks, always trace answers back to the original source passages.
  • Treating it as a Magic Bullet: Graph RAG is a tool for specific problems. Before implementing it, always compare its performance against simpler vector and hybrid search baselines. The original paper's reported improvements were on global sensemaking questions over datasets of around one million tokens; these results may not generalize to all query types (Source). Many RAG failure modes are not solved by graphs alone, and it's essential to have strong RAG evaluation metrics in place.

Understanding the Full Cost of a Graph RAG System

The biggest drawback of Graph RAG is its high upfront and maintenance cost, which is significantly greater than that of a standard vector RAG system. Before committing to this architecture, it is critical to understand where these costs come from.

The total cost can be broken down into three main categories:

  1. LLM Extraction Calls: This is often the largest single cost during the indexing phase. You must process every document in your corpus with a powerful LLM to extract all entities and relationships. For large document sets, this can translate to a significant number of LLM calls and substantial expense.
  2. Graph Construction and Storage: After extraction, you need the compute power to run community detection algorithms and construct the graph. You also need to pay for storing the graph itself, which includes nodes, edges, properties, community structures, and summaries. This may require a specialized graph database.
  3. Community Summarization: Once communities are identified, another round of LLM calls is required to generate summaries for each community, often at multiple levels of a hierarchy. This adds another layer of processing cost to the indexing pipeline.

Unlike vector RAG, where indexing is a relatively fast process of chunking and embedding, Graph RAG's indexing is a multi-stage, computationally heavy pipeline. This trade-off of high initial cost for the ability to answer more complex queries is the central decision you must make when considering Graph RAG.

What to learn next

Sources and further reading