RAG, GraphRAG, and what we build instead

Ask an AI vendor how they’ll answer questions from your documents, and three answers tend to come back: retrieval-augmented generation, a knowledge graph built by an LLM at index time, or something described as both at once. In sales conversations the names blur together. Mechanically, they are not the same system, and the difference is not a matter of taste, it decides which answers you can trust.

Three ways to answer the same question

Vector RAG chunks your documents, embeds the chunks, and at query time retrieves whichever pieces sit closest to the question in embedding space. GraphRAG goes a step further: it prompts an LLM to extract an entity-and-relationship graph from the corpus, groups related entities into communities, and searches over that structure instead of flat text. We build a third thing: an OWL ontology, defined from the questions the business needs answered, populated with entities resolved against real-world identity, queried through SPARQL. All three read the same documents. Where they stop looking alike is what each one hands back when you ask the same question twice.

Where similarity search fails silently

Take a real example: is the fire door on the fourth floor rated T30, and which document proves it? Ask a vector RAG system and you can get three different answers depending on which chunk happened to embed closest that run: a fire-safety report says T30, a delivery note nearby says T0, an unrelated paragraph gets pulled in because it uses similar vocabulary. Ask again five minutes later and the ranking can shift enough to change the answer. Nothing is broken; the system is doing exactly what nearest-neighbour search does. “Closest in embedding space” and “true” are two different properties, and only one of them is what the question needs.

A knowledge graph built over the same documents returns one answer: door T-04-117 is documented T30 in the fire-safety report, page 14, cross-referenced to IFC element #4471, and it also surfaces that a nearby delivery note contradicts it, because the graph keeps both facts and both sources instead of discarding the contradiction. Ask it once or a hundred times, the traversal is identical, so the answer is identical.

What GraphRAG gets right, and where it stops

Graph-based RAG is a genuine improvement on flat RAG, not a rebrand of it. Extracting entities and relationships into a graph, then summarizing communities of related entities for high-level “global” queries alongside entity-anchored “local” search, measurably beats chunk retrieval on multi-hop questions, the kind where the answer requires connecting two or three documents rather than finding the single closest one.

The limit is what the graph is built from. It’s extracted by an LLM, fresh, on every indexing run, from whatever the model happens to notice in the text that time. There’s no ontology constraining what a class or a relationship is allowed to mean, no entity-resolution guarantee that the same supplier named in five documents becomes one node instead of five, and no version history: re-index the same corpus and the resulting graph can be structurally different, with no record of what changed or why. It inherits the extraction step’s uncertainty without a way to express it.

What “governed” means in practice

The difference is a property you can test, not a claim you have to take on faith. Ask the same question of a governed graph twice, a year apart, and you get the same answer traced to the same source, or you get a change you can point to, an updated document, a new ontology version, not silent drift. Entity resolution happens once, as a modeled step, so “Müller GmbH” and “Müller GmbH, Niederlassung Stuttgart” collapse into one node with a defined relationship instead of staying two unconnected mentions. Every fact carries where it came from: the document, the page, and how confident the extraction was.

Where each tool belongs

None of this makes vector RAG or GraphRAG the wrong tool. For “summarize what changed in this contract” or “find documents like this one” across a messy, unstructured corpus, at zero modeling cost, they’re the right one. What they’re not built for is “list every machine that contains this recalled part, and who to call about each one” or “has this door always been rated T30, and who confirmed it”: questions that need to be exact, complete, and answered the same way on the tenth ask as on the first. That’s a routing decision between two genuinely different tools, not a claim that one replaces the other.

On a comparable structured-query benchmark, GPT-4 answering directly from a raw schema reached 16% accuracy. The same model, answering from a knowledge-graph representation of the identical data, reached 54% (Sequeda et al., 2023). The model was the same; the difference was the representation it was allowed to stand on.