Provenance by construction, not by afterthought
Extraction is LLM-assisted. Storage and querying are not. Saying which is which, and proving it per fact, is the difference between “auditable” and “trust us.”
Every serious knowledge-graph vendor eventually says some version of “zero hallucinations.” The honest version of that claim is narrower, and it is worth saying plainly before a skeptical reader has to ask: extraction, the step that turns a document into a fact, is LLM-assisted. Storage and querying, what happens after a fact is in the graph, is not. Confusing the two is where the marketing usually goes wrong, and keeping them apart is the engineering’s job.
Say the uncomfortable thing first
An LLM reads a document and proposes a fact. That step inherits everything an LLM inherits: it can miss a qualifier, misjudge which clause modifies which noun, or flatten a hedge into a certainty. Naming this upfront is what makes the rest of the argument credible. A vendor claiming their extraction never does this is the one a technical buyer should trust less, not more, because it is not a claim anyone can verify.
What gets lost in extraction
Take the kind of message a corporate email archive is full of, the Enron corpus, still the standard public benchmark for exactly this problem, is a good illustration: “the deal is basically done, pending legal sign-off.” A clean extraction reads that as one fact: deal approved. A careful one reads two: a status, and a condition attached to that status. Compress the sentence to “Deal 7743: approved” and the qualifier, the part that matters most if legal never signs off, is gone. Nothing in the resulting triple looks wrong. It just is not complete, and a triple store has no native way to flag its own incompleteness.
What provenance buys you
The fix is not better extraction, extraction will always have an error rate. The fix is that every fact remembers exactly where it came from: which message, from whom, at what time, and how confident the extraction step was in reading it that way.
ex:fact-001
a rdf:Statement ;
rdf:subject ex:Deal_7743 ;
rdf:predicate ex:status ;
rdf:object "approved, pending legal sign-off" ;
prov:wasDerivedFrom <mailto:msg-19077@enron-corp.example> ;
prov:generatedAtTime "2001-09-14T11:02:00Z"^^xsd:dateTime ;
ex:confidence 0.71 .None of this is exotic, it is standard PROV-O reification, the same vocabulary any RDF store can read without our involvement. What it buys is simple: if “legal sign-off” turns out to matter, the citation is one click from the answer to the original message, not a leap of faith in a black box.
What happens when confidence is low
A fact extracted from unambiguous text and a fact extracted from a hedged, informal one are not interchangeable, and a graph that stores them identically is lying by omission. Low-confidence extractions get flagged rather than presented with the same authority as a confirmed filing. The flag doesn’t remove the uncertainty; it keeps it visible.
One claim you can prove, one you can’t
“Auditable” is a claim we can prove on demand: show the message, show the timestamp, show the confidence. “Zero hallucination” is not a claim anyone can prove, about any system, ever. It is worth asking any vendor which of the two they are making.
More from the blog