Why not just pick a 0.9 threshold by hand and skip the statistics?

You can, and the README prices both routes. Hand-picking 0.9 auto-routes 82.25% at $0.89 per 1k and 0.99 auto-routes 70.50% at $1.38, while the certified 5% threshold auto-routes 84.75% at $0.79. What you cannot get by hand is the bound: the difference between a measured 0.0225 and a certified 0.0499 is a guarantee, not an observation.

How is this different from fitting a threshold and checking it on a held-out split?

The README's own comparison says a tool like jevcal fits a per-question threshold to a target accuracy and verifies it on a held-out split, which is an estimate. Conformal risk control instead returns a finite-sample bound on the share of incoming queries silently routed to the wrong handler. Whether that distinction matters is a question about your risk tolerance, not about accuracy.

The cheaper-than-an-LLM claim — cheaper than what, and is the fallback any good?

Cheaper than always escalating to an LLM: $0.1455 per 1k queries for Jev at p50 0.37 s and p95 0.744 s, against an assumed $4.20 per 1k for escalation. The README leaves the always-LLM accuracy cell blank because it never measured the fallback model, so compare that row on price only, not on quality.

What actually breaks the guarantee in production?

Three measured cases: prevalence shift at 3.6x the certified bound, out-of-scope traffic arriving mid-flight at 0.23 loss per query, and a 20-intent deployment meeting the other 130 intents at 1.0000 loss per query. The scope gate also fails to hold at a 5% budget, accepting 38.67% of the human out-of-scope set.

What dataset and ground truth does jev-certify use?

CLINC150: 150 intents, 4,500 human-labelled in-scope queries, and 1,000 human-written out-of-scope queries. Ground truth is human, so nothing is graded by the model being graded.

Why is a 1% risk target infeasible?

Jev returns exactly 1.0 on 56.4% of answers and 9 of those are wrong, so the attainable bound floors out at 1.95%. The certificate's resolution floor is the model's own probability rounding, not the maths.

How many hand labels does the production audit need?

Forty labels beat what 254 labels achieve with labels alone, a 62% narrower interval. Past about 100 labels the limit becomes served traffic, not labelling budget, so a two-point target is unreachable at any budget on 550 decisions.

CommunityJev Practices & Patterns

Jev Certify

jev-certify turns Jev's probabilities into a bounded claim: conformal risk control picks a routing threshold with a finite-sample guarantee on silently misrouted queries, and prediction-powered inference audits it with about 40 hand labels. Measured on 2,412 Jev 1.13 decisions over CLINC150.

See highlights

Open github.com

Stars: 1
Language: Python
License: MIT
Published by: Community
Author: nikkoxgonzales
Last verified: Oct 2026

nikkoxgonzales/jev-certify

User repository on GitHub

View on GitHub

Finite-sample guarantees for Jev (TypeSafe's System One). Conformal risk control turns calibrated probabilities into certified routing thresholds; prediction-powered inference audits them. 2,412 decisions on CLINC150 for $0.23 — including the shift and prevalence cases where the guarantee breaks.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Turning Jev's calibrated probabilities into claims that survive an audit. TypeSafe's Jev returns calibrated probabilities, not guarantees. A probability tells you what the model thinks; it does not tell you what your system will do, and a confidence threshold picked by taste has no idea what its own error rate is. This project adds the missing layer:

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification Jev for Routing Jev for Tool calling

Used inJev for Software & DevTools

Languages: Python

Highlights

At a 5% risk target Jev settles 84.75% of traffic with 2.65% error among settled, and it held out of sample: 0.0225 measured, 0.0499 certified.
A 1% target is infeasible: Jev returns exactly 1.0 on 56.4% of answers and nine of those are wrong, flooring the attainable bound at 1.95%.
40 hand labels beat 254 with labels alone for a ±0.05 target, a 62% narrower interval; ±2pp is unreachable at any budget because traffic is the limit.
Three measured breaks: prevalence shift ran 3.6x the certified bound, out-of-scope traffic cost 0.23 per query, a 20-intent deployment lost 1.0000 per query.

Watch out

Everything is measured on one dataset: CLINC150 with 150 intents, 460 calibration queries and 400 held out, where Jev's own top-1 routing accuracy is 93.0%, so the certificate describes this traffic and not yours. The always-escalate-to-LLM row is deliberately blank on accuracy because the fallback model was never measured, and its $4.20 per 1k price is the only assumed constant in the cost table, so the 81% saving claim is a price comparison rather than a quality-adjusted one. The scope gate holds at a 1% budget and fails at 5%, where 38.67% of human out-of-scope queries are accepted. Calibration is uneven by question: the noul probability has ECE 0.169 and Brier 0.084 against the router top probability at ECE 0.099 and Brier 0.106. It is built against Jev 1.13 through the OpenRouter Decisions API at POST /api/alpha/decisions, and Jev is not a chat model, so chat-completion SDKs cannot reach it. MIT licensed with 43 tests passing, and the positioning against other threshold-fitting tools is the author's own reading of a 268-entry catalog, not an independent review. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

dejevu

Python browser agent that reads one lean page view per state and answers with a single JSON action, using any OpenAI-compatible model with no decision API or browser daemon.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

idovmamane/dejevuPython

daf-jev

Python client and decision toolkit for the Jev System One endpoint with noul, choice, and score primitives, batched asks, composition helpers, concurrent evaluation, and an MCP server.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

docxology/daf-jevPython

siftr

Four read-only MCP tools that rank files, cut a file down to the relevant part and pick from long lists, measured on a frozen SWE-bench Lite split.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

Bentlybro/siftrPython

jev-mode

Python library and CLI that moves bulk semantic judgments — ticket triage, stale-file tagging, bug or question routing — out of the agent's context into typed Jev questions, with a measured comparison against an in-context control.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

ddfeyes/jev-modePython

jev-routing-experiment

Runs Jev as an LLM router over two public benchmarks, with a no-Jev ablation and RouterArena's own scorer so the numbers sit next to the public leaderboard.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

TokenTrim/jev-routing-experimentPython

jev-skillbench

Measures whether Jev can catch malicious agent skills on MalSkillBench, with a cascade arm that sends only the uncertain band to an LLM. Per-sample records, thresholds and calibration are all kept.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

MohibShaikh/jev-skillbenchPython

From the Jev guides

Original write-ups on what this entry is for.

Designing the State: What to Include and What to Drop

Oct 12, 2026

Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.

Jev Practices & Patterns#state#design#primitives

Read article

Noul, End to End: One Probability and a Dozen Thresholds

Oct 12, 2026

A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.

Jev Practices & Patterns#noul#primitives#confidence

Read article

Scoring Without Measuring: Bands, Rubrics, and Composite Triage

Oct 12, 2026

A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.

Jev Practices & Patterns#score#primitives#scoring

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai