What 12,727 emails taught us about graphs
We put four ways of answering questions from documents on the same corpus and the same questions. A lexical method from 1994 won. Why we publish that, and why the graph we build derives its schema from questions.
Every vendor in our field says their retrieval is better. Few show a table. This is ours, and it starts with a result we would have preferred not to get.
The corpus is public: the Enron email archive, the standard benchmark for question answering over private company mail, because it is the only large body of real corporate correspondence anyone is allowed to use. We deduplicated it, since half of the 517,401 messages are the same mail filed in several folders, and took the first five percent by date: 12,727 emails. The questions come from EnronQA, a published set of over half a million question and answer pairs written against those emails. 4,783 of them are answerable from our slice, and every engine got exactly those.
Four engines, one answerer
The four. BM25, a lexical index from 1994 with no model in it. Dense , the embedding index most products mean when they say AI search. AutoSchemaKG with HippoRAG, an open triple extraction with PageRank retrieval over the resulting graph. And Microsoft GraphRAG. Every engine hands its retrieved context to the same answering model with the same character budget, so the chart measures retrieval and nothing else. Alongside them ran an early -typed graph of our own, in which a language model may extract only what an allows. It stays out of the chart: it was a first attempt on a schema that did not fit this corpus, and what it taught is further down.
BM25 wins on both metrics, costs nothing to build and answers in 32 milliseconds. Dense follows. Both graphs are behind them.
Why we publish it anyway
Because the number is a property of the question, and knowing that is worth more than a flattering chart. EnronQA questions were generated from their source email, so they inherit its vocabulary: names, tickers, deal numbers. Single hop, single inbox. That is the ideal case for matching words, and it is not what a graph is for. A graph earns its cost when the answer requires connecting two documents that share no vocabulary, when “who approved this” has to follow a chain from a memo to a name to a role. The public multi-hop set for Enron, ConcurrentQA, holds 400 email-only questions. Zero of them have both hops inside our five percent slice. The axis on which the graph should win was not in the test. We have since built the corpus that contains it, 46,151 messages defined by the evidence the questions cite rather than by a date cut, and that run is next.
The second reason is discipline. If a 1994 baseline beats your engine on a task, every future engine has to clear it before it is called an improvement. That bar is now in our harness, permanently.
What the graph told us about itself
The scores hide the more interesting finding, which is structural. The open extraction produced a graph of 236,053 nodes and 355,914 edges, with 65 percent of the nodes in one connected component: organically wired, because open triples connect almost any two entities that share a sentence. Our early -typed extraction, run on the same emails, produced the opposite shape. Most of its nodes had no direct edge at all. An written for construction found almost nothing to type in the administrative email of an energy trader in early 2000, and a graph with no edges cannot be walked. That is not a verdict on either approach. It is a measurement of what happens when the schema and the corpus do not belong together.
Where the schema has to come from
That measurement is the design brief for the pipeline we build now. The schema is not written by us and not guessed by a model. It is derived from questions. Every regulatory document a project has to satisfy, DIN, HOAI, the state building code, is read clause by clause and turned into the questions that clause would ask of a building. The engineers add the questions they actually put to their own project. Together that is one approved set of competency questions, and the is built from it question by question, each addition checked against the questions it is meant to answer. Requirements become constraints the graph can enforce. Only then does extraction begin, and a model may extract only what that allows, checked as each fact lands.
The other half is that the same answers the questions. A question that matches one of the competency questions runs on a prepared, deterministic path and never touches a model. For any other question, the says whether a path between the things asked about exists at all before anything is generated; if it does not, no amount of prompting will produce an answer, and the system says so instead of guessing. Ranking over text is the fallback for what the cannot express, not the default.
This is why we expect it to hold where the graphs above did not. The vocabulary mismatch that emptied our early graph cannot occur when the schema is derived from the corpus’s own questions. Every fact is typed on a path the query side knows, so the graph is walkable by construction. And an answer to a competency question is the same answer every time, with the clause and the page it came from. Whether that shows up as a number is what the multi-hop run is for, and it will be measured against the same lexical baseline as everything else.
The bug the numbers confessed
One more thing the benchmark did that a demo never would. That early build attributed every extracted fact to all five emails it was reading at the time, not to the one it came from. Nobody saw it in the code. The graph showed it: 99.94 percent of nodes had a mention count divisible by five. Once the provenance was repaired, without re-extracting anything, retrieval recall roughly doubled on a subsample. A fact that names the wrong source is worse than no fact, and the only reason it was found is that we counted.
What we take from it
Three rules, all now in force. Measure against the cheapest thing that could possibly work. Never put two numbers in one table unless corpus, harness and scorer are the same. And treat the structure of the graph as a result in its own right, because it tells you whether the schema fits the data before a single question is asked.
The multi-hop run will say whether a graph wins where it is supposed to. Until then the honest sentence is: on single-hop email, a graph does not buy retrieval quality, and we do not sell it as if it did. What a graph grounded in its own questions buys is the answer that names its clause and its page, holds still when asked twice, and can be walked when the question spans documents. That is a different product, and it is the one we build.
Data: EnronQA (arXiv:2505.00263) and ConcurrentQA (arXiv:2203.11027), both public. Corpus definitions, run records and the full per-engine tables are kept in our lab notes with commit and date; the numbers here are from the run of July 2026.
More from the blog