My question works when I try it by hand. Why calibrate?

Hand-testing hides the failure shape. The repository's own first draft got 18 of 26 right, with four wrong answers at 0.94-plus confidence. Calibration runs your questions over labelled examples to find that out.

What do I do with a ranker verdict?

Do not cut on it. If the report suggests a threshold that hits your targets, set it and re-check; otherwise sort by probability and review from the top.

How much does a calibration run cost?

Almost nothing. The bundled 66-message example checks in about 8 seconds for a fraction of a cent, against TypeSafe, OpenRouter, or any server with the same request format.

What files do I actually write?

questions.json for the questions plus each operating point, and examples.jsonl with one labelled example per line. jev-calibrate init scaffolds both and never overwrites existing files.

Which providers can jev-calibrate talk to?

The TypeSafe API, OpenRouter, or any server that implements the same request format. Set one key, TYPESAFE_API_KEY or OPENROUTER_API_KEY; when both are present TypeSafe is used unless you pass --provider openrouter.

What is the recommended tune-to-holdout procedure?

Run lint until it reports no errors, then check the tune half. Read the misses and edit only the criteria, check again and run compare, which exits 1 when anything regressed. When the tune half looks right, stop editing and run check --split holdout once; that number is the one to report.

How does the tool keep holdout results honest?

check --split holdout appends what it judged to .jev-calibrate/ledger.jsonl with the question, its revision and short hashes of the examples. A revision hashes the question text plus its decisions entry, so moving a threshold counts too. A later run warns how many holdout examples already judged an earlier revision.

CommunityJev Practices & Patterns

Jev Calibrate

CLI that calibrates Jev questions against your own labels: write questions, label examples in JSONL, run check, and get a verdict per question — gate, gate-above-confidence, ranker, unusable, or too-few-examples.

See highlights

Open github.com

Stars: 32
Language: TypeScript
License: MIT
Published by: Community
Author: smkrv
Last verified: Oct 2026

smkrv/jev-calibrate

User repository on GitHub

View on GitHub

Calibrate Jev questions against your own labels: tune criteria on labelled examples, confirm on a held-out set, get a verdict per question. Unofficial.

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Calibrate Jev questions against your own labels.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

The bundled example checks 66 labelled support messages in about 8 seconds for a fraction of a cent.
Verdicts name the operating point: act on the answer, act above a confidence cut, sort by probability, or rewrite the question.
TypeScript with no runtime dependencies; talks to TypeSafe, OpenRouter, or any server with the same request format.
jev-calibrate init scaffolds questions.json and examples.jsonl and never overwrites existing files. MIT licensed.

Watch out

Unofficial and unaffiliated with TypeSafe; verdicts measure the examples you labelled, not production traffic. Copy reviewed against the repo README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-lm

Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.