A citation can point at a real document and still be wrong. The quote may not exist, the section may discuss the topic without supporting the claim, or the number may be correct while the cited row is the wrong one. Verification is therefore several checks stacked together, and only some of them need a model.
The useful split is: prove the quote exists, classify the relation between the passage and the claim, and keep numeric and role checks in code. Everything below is a variation on that split.
The quote check belongs in code
The citation-check cookbook starts with an exact string match after normalizing whitespace and curly quotes. If the answer's quote cannot be found in the cited section, the verdict is fabricated before any model call. That is the cheapest possible check and the one that catches the most obvious failure: an answer that invents a quotation and attaches it to a real source.
The cookbook runs over RFC 7519, split into 45 numbered sections, with eight citations from an LLM answer: four accurate, one fabricated, one contradicted, and two unsupported. All four accurate citations came back verified at confidence 0.93 or higher, and all four planted failures were caught. The caveat is important: because the match is exact after normalization, a truncated or lightly reworded quote is also reported as fabricated. That is a conservative failure, but it is still a failure a person has to resolve.
One Choice for the relation
If the quote exists, the remaining question is whether the section supports the claim. The cookbook asks a single Choice with three options: supports, contradicts, and says_nothing. Code maps that answer plus the string-match result into four verdicts: verified, unsupported, contradicted, fabricated.
The contradicted example scored 0.99. That is the strongest argument for asking the relation question rather than trusting confidence alone. A high-confidence answer can be confidently contradicting the claim. Meanwhile the two unsupported examples scored 0.27 and 0.56 and fell below the default 0.8 auto-accept gate, which is how a confidence gate should behave: the score decides whether a human looks, not whether the claim is true.
The fact that Noul carries no separate confidence also matters here. A Noul returns one probability, so if you want a second axis for review, you either use a Choice or build a calibrated threshold on the probability itself.
Documents that need more than a quote
The Paper Trellis Citation Verifier applies the same idea to biomedical manuscripts, with a useful extra constraint. Claude proposes a verdict and a verbatim evidence passage, and the application discards any quote that cannot be found in the fetched paper text. Jev then reads the citing sentence beside the proven passage and returns probabilities for supports, contradicts, or says nothing. Model output never overwrites a human verdict.
Its handling of missing evidence is the part worth copying. Paywalled papers are checked against the abstract only, and an abstract that is silent on the claim is reported as needs full text, never as unsupported. Scanned PDFs are not read at all. That distinction matters in any document pipeline: absence of evidence is not evidence against the claim, and a checker that confuses the two will send good citations to review.
The project is MIT-licensed and runs in the browser with no database or application server. The trade-offs are equally clear: bring-your-own Anthropic and TypeSafe keys, citing sentences and paper text leave the browser for those providers, and the README says the thresholds have not been validated on a labeled biomedical set. Treat the probabilities as triage.
Verification is not reranking
It is tempting to reuse a reranker for citation checking. The rerank cookbook shows why the two jobs are different. Over 3,565 CLERC passages and 40 legal queries, BM25 put the gold passage in the shortlist 100% of the time but at rank 1 only 5%. One Noul per query-candidate pair lifted top-1 to 18% and top-10 from 38% to 62%, using 1,200 calls and 1,536,002 input tokens for $0.0645. Reranking reorders candidates; it cannot recover a passage the shortlist missed. Citation verification asks a stricter question about one passage and one claim, and it must be able to say "contradicts" rather than merely "less relevant."
There is also a warning about assuming decision models are automatically better rankers. An independent arXiv study, System One Models Are Not Better Rerankers, compared decision fine-tuning with reranker fine-tuning across six Qwen backbones from 0.6B to 27B and ten TREC DL and BEIR collections. From 0.6B to 9B, reranker fine-tuning won on every backbone by 5 to 29 points nDCG@10; at 27B the two approaches tied. Up to 9B, decision fine-tuning added nothing over asking the untuned model. That is a study of fine-tuned open models, not of hosted Jev, but it is the right prior: use the tool that was trained for the ranking job.
Numbers can hide role errors
A financial report makes the citation problem harder because the same number appears in several places. The same-number citation swaps study holds operands and arithmetic fixed while moving citations between cells with identical values. The result is a class of error that numeric matching cannot see: a calculation that is numerically correct while citing the wrong financial role.
The paper reports that a signed-number-at-pointer baseline explains most of the recovery over exact quotation checks, leaving a smaller role-recognition problem for probabilistic verification. Explicit column labels improved selected wrong-role decisions while lowering support for some equivalent evidence. A follow-up on 36 new source pages, labeled by a single non-author reviewer, exposed the same tradeoff. This is an arXiv preprint in a controlled setting, not a deployed financial assistant, but the lesson transfers: a verifier has to see the label and the role, not only the digits.
Retrieval and citation are different gates
Before a citation can be checked, the passage has to be retrieved and filtered. The RAG passage cookbook asks four Nouls per passage (relevance, evidence, contradiction, and prompt injection) and routes in code with thresholds in one dictionary. A planted injection was dropped at 0.99, and a false-premise query kept its contradicting passage at 0.92. The caveat is explicit: the injection question is a filter, not a security boundary, because passages below threshold still reach the prompt.
At the other end, Vector Graph RAG uses Jev to select graph relations and rerank full passages alongside vector-search candidates. The two-stage configuration reached 86.07% average Recall@5 over 1,000 MuSiQue and 1,000 2Wiki queries, at the cost of 3.13 seconds of model-call latency. Better retrieval improves the inputs to verification, but it does not replace the support check.
What to put in the gate
The 130k claim-verification dataset is a useful evaluation corpus rather than a policy. It pairs RAG claim-verification and response-quality questions with jev-1.13.0 predictions, gold labels, confidences, token counts, latency, and cost: 126k train, 2.09k validation, and 2.06k test rows derived from RAGTruth. The license is listed as other, with a source license of mit-with-upstream-review-required, so check the upstream terms before redistributing.
Two guardrails from the jaggedness list apply to every checker. Complementary probabilities do not always sum to one, so do not build a policy on the assumption that supports and contradicts are exact complements. And the model answers the words you wrote, so the relation criteria need boundary cases rather than a topic label.
A workable gate is: string-match first, relation Choice second, numeric and role checks in code, and human review for anything below threshold or marked needs full text. The pipeline is slower than one confident answer, but it fails in places you can see. The retrieval article covers filtering and reranking; the measurement article covers evaluation. This gate decides whether an answer keeps its citation.
