Docs
OfficialJev Cookbooks
Cookbook: Self-Consistency for Choices
A repeatability study: an 8-question moderation rubric run 15 times per condition shows a mean probability standard deviation of 0.0098, and a 0.60 uncertainty gate lifts agreement to 99.2%.
- Format
- Docs
- Reading time
- 3 min
- Published by
- TypeSafe AI
- Last verified
- Sep 2026
Used inJev for Software & DevTools
Highlights
- 15 repeats across 9 conditions over 8 Choice questions, with jev-1.13.0 returning in 114 ms at $0.000046 per call.
- TypeSafe's mean probability standard deviation was 0.0098 versus 0.0245 to 0.0543 for five LLM distributions.
- With top probability below 0.60 routed to uncertain, policy agreement reached 99.2% while 74.2% of labels stayed automatic.
- TypeSafe still flipped its top label on primary_risk and link_handling across its own repeats.
Quickstart
python
MIN_CHOICE_PROBABILITY = 0.60 # top label below this -> "uncertain", human review
questions = {key: Choice(instructions=instructions, criteria=choices)
for key, (instructions, choices) in QUESTIONS.items()} # 8-question rubric
response = typesafe_client.system_one(
model="jev-latest",
state={"uid": f"{RUBRIC_HASH}:{sample_index}:{token_hex(4)}", "post": POST},
questions=questions,
)
for key in QUESTIONS: # repeated 15x; did the label hold still?
probs = response.answers[key].probabilities
top = max(probs.values())
selected = max(probs, key=probs.get) if top >= MIN_CHOICE_PROBABILITY else "uncertain"Watch out
These are repeatability numbers, not accuracy numbers, and a deterministic temperature-0 baseline can score 100% agreement by never abstaining.
Questions about this entry
Short answers drawn from the linked project or page.
15 repeats across 9 conditions over 8 Choice questions: a mean probability standard deviation of 0.0098 for jev-1.13.0, against 0.0245 to 0.0543 for five LLM distributions. These are repeatability numbers, not accuracy numbers.
When the top probability falls below 0.60 the label becomes uncertain for review. The cookbook reports policy agreement reaching 99.2% while 74.2% of labels stayed automatic, and notes TypeSafe still flipped its top label on two questions across its own repeats.
One user post built to sit on the fence: heated and insulting language aimed partly at a person and partly at the community, an off-platform invite linking to discord.gg, one prior strike on the account, four user reports for harassment, spam and threat, and threat-like wording that is never cleanly phrased. Small wording differences should not randomly move the same post between enforcement paths.
Nine conditions. The non-reasoning claude-haiku-4-5 and gpt-5.4-mini at temperature 0 and at the API default, the reasoning gpt-5.5 and claude-opus-4-8 which have no temperature dial, and TypeSafe's one system_one call over the 8 Choice questions with a fresh throwaway uid on each call. Each run answers all 8 questions in a single call, and each condition is repeated 15 times.
The LLM distribution settings repeated their plurality labels 87.5% to 100% of the time against TypeSafe's 90.8%, and TypeSafe had lower mean probability variation than five of the six LLM distribution conditions, with Haiku at temperature 0 varying less. Close probabilities still permit routing changes: TypeSafe still flipped on 2 of the 8 questions, primary_risk and link_handling.
jev-1.13.0 returned in 114 ms at $0.000046 per call at the historical rate of $0.042 per million input tokens with output free. The comparison run uses jev-latest on the production API, sampled on 2026-09-11, and the LLM prices used in the notebook are the July 2026 figures for each model.
Because agreement alone can be bought by never abstaining. The cookbook's own caveat notes that a deterministic temperature-0 baseline can score 100% agreement while taking no uncertain outcome at all, which is why the gate is reported next to the share of labels that stayed automatic.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Building a contextual brand-safety scorer on Jev from scratch.
X: A website slop detector that checks 35 tells in 243 ms
X: Breaking, golden nugget, or slop
X: A filter for reply-guy comments
More Jev resources like this
A 14-Noul claims-triage rubric over one insurance claim, repeated 15 times: mean probability standard deviation of 0.0102 at 111 ms per call, with a 0.30 to 0.70 band for human review.
Jev CookbooksDocs#official#cookbook#consistency
Official
Score retrieved passages with four Noul questions each, then route in code: prompt injections dropped, false premises kept as conflicts, and evidence assembled separately for the generator.
Jev CookbooksDocs#official#cookbook#search
Official
Turn wine tasting notes into 67 numeric columns for a CatBoost regressor: five rounds of reading its own errors cut held-out RMSE from 3.09 to 1.77 across 800 unseen reviews.
Jev CookbooksDocs#official#cookbook#research
Official
Screen every message going into and out of an LLM app: a Noul describes possible hazards, a Score rates severity, and code thresholds the probabilities.
Jev CookbooksDocs#official#cookbook#guardrails
Official
A live chat filter that asks three questions per message — category, toxicity, and spoilers — then turns the answers into show, highlight, dim, hide, or blur in plain code. The 29-message example hides seven, blurs two, and highlights six.
Jev CookbooksDocs#official#cookbook#moderation
Official
Semantic search over GitHub's Terms of Service: one request ranks all 218 lines with a Choice while a Noul checks whether the document contains an answer at all, including when it should say no.
Jev CookbooksDocs#official#cookbook#search
Official
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.