Why a graph can count and a chatbot can’t

Ask a plant manager which delivered machines contain a recalled part, and the question sounds simple. Mechanically, it is not a retrieval question at all, it is a set-membership question: return everything that matches, and prove there is nothing left out. Vector search was never built to answer that, and two recent benchmarks now measure exactly how badly it fails.

A ranked list has no notion of “all”

A vector index returns the top-k passages closest to a query embedding, a fixed-size ranked list, by design. That is the right behaviour for “find documents like this one.” It is the wrong behaviour for “find every machine that contains part V-220”: if the true answer set has 47 members and k is 10, the system does not know it is missing 37 of them, and neither does the user reading a confident-sounding answer built from 10 passages. High recall at a fixed k still isn’t completeness, and completeness is the whole point of the question.

The literature already measured this

A 2026 benchmark built specifically to test this, AGGBench, formalizes aggregation queries as needing to “find all,” not “find one,” and reports recall scores under 0.05 for retrieval-based approaches on exactly this task, both Text-to-SQL and RAG fail to reach completeness, not by a small margin. A separate benchmark, GlobalQA, isolates counting and corpus-wide reasoning specifically and scores it directly against a graph-structured method:

Neither number is high in absolute terms, this is a genuinely hard task class. But the gap is over four-fold with the backbone model, the questions and the corpus held constant. It is the same mechanical limit this post opened with, now measured. Even the GraphRAG paper (Edge et al., 2024) makes the same point from the other direction: it built local/global search specifically because “global questions… are inherently a query-focused summarization task, rather than an explicit retrieval task,” one flat RAG was never built for. And the underlying reasoning gap isn’t new: HotpotQA (Yang et al., 2018) was already showing that questions requiring multiple linked facts break single-shot retrieval, eight years before either of these newer benchmarks confirmed it holds for aggregation too.

Why this isn’t a prompting problem

The instinct is to fix this with a better prompt: “list all of them, be thorough.” That doesn’t reach the actual bottleneck. The model only ever sees what was retrieved before it started generating, so no instruction inside the prompt can recover the 37 machines that never made it into context. Ask the same question twice and the ranking can shift enough to return a different 10, which means the system isn’t just incomplete, it’s incompletely different each time. That’s a retrieval-layer problem, and it has to be fixed at the retrieval layer.

What a graph does differently

A SPARQL query over such a graph isn’t a ranked shortlist, it’s a traversal: return every node that matches the pattern, however many there are. Ask “which machines contain V-220” and the answer is the join of the bill-of-materials graph with delivery records, all 47 rows or none, with a query plan you can inspect if the count looks wrong. None of this makes vector search obsolete, “summarize the recall notice” is still a good RAG question. But “list every affected machine” was never a ranking problem to begin with, and no amount of better retrieval tuning changes what kind of question it is.