Docs
OfficialJev Cookbooks
Cookbook: Double-Checking Citations
Catch wrong and hallucinated citations against RFC 7519 with a string match plus one Choice question: four accurate citations verified at 0.93 or more and all four planted failures caught.
- Format
- Docs
- Reading time
- 2 min
- Published by
- TypeSafe AI
- Last verified
- Sep 2026
Highlights
- Eight citations across RFC 7519's 45 sections: four verified at confidence 0.93+, one fabricated, one contradicted, two unsupported.
- An exact string match catches missing quotes with no model call, while the contradicted example still scored 0.99.
- Unsupported citations at 0.27 and 0.56 fell below the 0.8 auto-accept gate and went to a human.
Quickstart
python
QUESTIONS = {
"relation": Choice(
instructions="How does the section relate to the claim?",
criteria={
"supports": "The section states the claim or directly implies it is true",
"contradicts": "The section states the opposite of the claim",
"says_nothing": "The section does not address what the claim asserts",
},
),
}
response = client.system_one(
state={"claim": claim, "section": section}, questions=QUESTIONS, model="jev-1.12"
)
# no quote in the source -> "fabricated"; AUTO_ACCEPT = 0.8 gates human reviewWatch out
The string match is exact after normalization, so truncated or lightly reworded quotes come back as fabricated.
Questions about this entry
Short answers drawn from the linked project or page.
An exact string match with no model call: after normalization it flags missing quotes immediately as fabricated. The Choice question then classifies the section's relation as supports, contradicts, or says nothing.
Below the 0.8 auto-accept gate. The unsupported examples scored 0.27 and 0.56 and went to review, while the contradicted example still scored 0.99 — which is why the relation question exists rather than a confidence check alone.
verified, unsupported, contradicted and fabricated, plus a confidence that flags the ones a person should look at. Fabricated is settled in code by the string match; the other three come out of the Choice question classifying the section as supporting, contradicting or saying nothing about the claim.
The source is RFC 7519 on JSON Web Token, fetched from rfc-editor.org and committed next to the cookbook as rfc7519.txt: 58,365 characters split into 45 numbered sections. Loading strips the page headers and footers that interrupt paragraphs, and the section splitter walks the RFC's own header lines, skipping appendix headers so an appendix only terminates the section before it.
The RFC hard-wraps its paragraphs and uses curly quotes, so a verbatim quote would not survive a raw substring search. The code normalizes whitespace and curly quotes first, then looks for the quote as a substring. The trade-off is that a truncated or lightly reworded quote comes back as fabricated rather than as a near miss.
pip install ipython 'cooksafe>=0.2.0,<0.3.0' and set TYPESAFE_API_KEY. Every call is cached to json_cache.json, which ships with the cookbook, so re-running replays the published results instead of calling the API; delete it to go live. The published figures came from jev-1.12 on 2026-08-16 with the auto-accept gate started high at 0.8.
Eight citations from an LLM answer about RFC 7519 went through the check: the four accurate ones came back verified at confidence 0.93 or higher, and all four planted failures were caught, a fabricated quote, a contradicted claim, and two unsupported citations sent to a human.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Agent Handoff Gate: agents verify what they hand off
X: Jev as an LLM guardrail inside Agentgateway
X: Vercel's fx safety reviewer, measured at 18× faster
Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.
More Jev resources like this
A two-stage extraction cascade: a mini model extracts, a Noul battery verifies each field in one request, and a 0.7 gate escalates to a reasoning model, sitting on the cost/quality frontier.
Jev CookbooksDocs#official#cookbook#extraction
Official
Semantic search over GitHub's Terms of Service: one request ranks all 218 lines with a Choice while a Noul checks whether the document contains an answer at all, including when it should say no.
Jev CookbooksDocs#official#cookbook#search
Official
A 14-Noul claims-triage rubric over one insurance claim, repeated 15 times: mean probability standard deviation of 0.0102 at 111 ms per call, with a 0.30 to 0.70 band for human review.
Jev CookbooksDocs#official#cookbook#consistency
Official
Score retrieved passages with four Noul questions each, then route in code: prompt injections dropped, false premises kept as conflicts, and evidence assembled separately for the generator.
Jev CookbooksDocs#official#cookbook#search
Official
A support-ticket cookbook that lets Jev decide which requests are worth remembering, then wraps each call so saved tickets ride along in the state. Six later messages route correctly with memory against two without.
Jev CookbooksDocs#official#cookbook#memory
Official
Decide which of 450 candidate product pairs from two beer catalogues describe the same entity: one three-level Score per pair splits 40 merges, 50 curator reviews, and 360 non-matches.
Jev CookbooksDocs#official#cookbook#entity-matching
Official
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.