Organization repository on GitHub
Does Jev's confidence mean what it says? Calibrating Jev (TypeSafe System One) with Platt scaling and isotonic regression.
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev calibration study
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
- Stars
- 3
- Language
- Python
- Reading time
- 3 min
- Published by
- Community
- Author
- AnthusAI
- Last verified
- Sep 2026
About this project
This is the follow-up to Making Decisions Instead of Generating Text, which ended with "test calibration against a held-out set." It reuses the dataset and calibration ideas from Classification-with-Confidence, where we squeezed confidence out of a local LLM's token log-probabilities. Here the model hands us the probabilities directly. Three ideas run through everything below: Accuracy is measured: the share of predictions that agreed with the ground-truth labels, over some set of examples.
Quoted from the project's README. Read the full README.
Used inJev for Software & DevTools
- Languages
- Python
- Capabilities
- Requires API key
Highlights
- On 8,801 unique sentiment examples, raw expected calibration error for P(positive) was 0.117.
- Platt scaling cut that to 0.052. Isotonic regression cut it to 0.008 on the held-out split.
- Isotonic matched or beat Platt at every calibration size from 20 to 5,280 examples.
- Noul accuracy rose from 47.6% at 50–60% confidence to 100% above 95% (2,104 of 2,104).
- For a two-option Choice, the confidence field matched 2 times p_top minus 1, within rounding.
Quickstart
bash
# results live in results/variants.json and results/raw_confidence_bands.jsonWatch out
No license file. One constructed sentiment set and jev-1.13.0. Neutral-tier labels are arbitrary by design. The whole dataset was about 330 input tokens per request and a median latency of 0.24 seconds.
Questions about this entry
Short answers drawn from the linked project or page.
Not safely. Raw expected calibration error for P(positive) was 0.117 on held-out data, so a stated 0.8 was not right 80% of the time. A two-parameter Platt curve cut it to 0.052 and isotonic regression to 0.008, because the miscalibration was not sigmoid-shaped. Isotonic matched or beat Platt at every calibration size from 20 to 5,280 examples.
No, and this is the trap. For a two-option Choice the confidence field is 2 times p_top minus 1, a measure of how peaked the distribution is rather than the probability the answer is right, and it carries no information beyond the top probability. Below 95% it is close to useless: Choice accuracy ran 50 to 57% at every stated confidence from 50% to 95%.
It holds, but it is narrower than it sounds. Across 8,801 unique sentiment examples, all 2,104 Noul answers above 95% were correct, and they are nearly all clear-cut: 978 strong tier and 1,083 medium, only 42 weak and 1 neutral. The weak spot is the middle, where 2,303 of 3,428 answers in the 55 to 80% band were weak or neutral tier.
Do not assume so. Everything comes from one constructed sentiment dataset whose neutral-tier labels are deliberately arbitrary, and one model version. The same setup scores two different ECEs depending on whether it measures P(positive) or confidence-in-the-predicted-answer, so the numbers are not interchangeable. Question wording also moved raw ECE from 0.064 to 0.160 across 11 setups.
About 330 input tokens per request across the whole dataset, roughly $0.12 at the published $42 per million input tokens with output tokens free, and a median latency of 0.24 seconds. There was no rate limiting at concurrency 8. Each example is sent as one request carrying several questions, which is where the small bill comes from.
Jev came out ahead of Llama 3.1-8B on the same 1,000 examples. Its confidence separated right answers from wrong ones at AUROC 0.83 against 0.72, and 0.90 against 0.76 once the neutral tier is dropped. The earlier project derived confidence from a local model's token log-probabilities; here the model returns the probabilities directly.
A constructed sentiment set of strong, medium, weak, and neutral tiers crossed with positive and negative, where the neutral labels are deliberately arbitrary as a domain-bias experiment. Exact repeats were dropped, leaving 8,801 unique examples split 60/40 with seed 42 into 5,280 calibration and 3,521 test. Results land in results/variants.json and results/raw_confidence_bands.json. There is no license file.
More Jev resources like this
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonA BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonA benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonA calibration write-up and scripts for Jev comment and code rules on a Unity project, with ordering.py to compare rule versions on real blocks.
Jev Benchmarks & Evaluations#community#evaluation#code-review
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.