Docs
CommunityJev Benchmarks & Evaluations
JevBench (Metamorphic Coherence)
Metamorphic coherence benchmark for typed probabilistic decision models: 240 synthetic cases and 2,240 questions checked by 50 probability laws, with released reports for 21 open models and 4 reference models.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- JevBench
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 240 synthetic English cases, 20 in each of 12 decision domains, with 2,240 questions: 1,341 Nouls, 615 Choices and 284 Scores; 302 golds are unknown.
- 50 metamorphic relations in five dimensions; every test edits only the question side and is frozen in advance.
- JevBench-240 runs 12,000 tests and 10,320 requests; the reported JevBench-mini runs 1,200 tests and 1,248 requests.
- The rewrite bank holds 823 accepted rewrites from 1,697 candidates, including paraphrases, option descriptions, Chinese translations and negations.
- Apache-2.0 for data and reports; cases were written by gpt-oss-20b and checked by Llama-3.1-8B-Instruct.
Quickstart
python
import jevbench as jb
model = jb.systemone(URL, model="my-model") # any server implementing the interface, or a Python function
report = jb.evaluate(model) # JevBench-mini
print(report.table("dimension"))Watch out
Card reviewed 2026-10-17; coherence is not correctness — the card warns that a model answering uniformly scores higher than every evaluated model, so coherence must be read beside accuracy and the Brier score. The public test set, synthetic English cases and model-based gold verification limit what any score proves.
Questions about this entry
Short answers drawn from the linked project or page.
A metamorphic coherence benchmark for typed probabilistic decision models: it asks whether probabilities stay coherent when questions are reworded, negated, combined, re-optioned or batched, using 240 synthetic cases and 2,240 questions.
Single laws of probability and choice in five dimensions: representation, batch independence, probability measure, logical order and choice-set conditioning, each rewriting the question while never editing the input state.
Directly: the questions are Nouls (yes/no), Choices over declared options and Scores on ordered scales, exactly the typed shapes a System One model answers with a distribution instead of text.
Coherence needs no gold labels — it checks laws over each model's own answers for the same input. Gold answers exist only for the accuracy reported beside it, alongside Brier and log loss on unchanged answers.
Fact-first generation: hand-written specifications compute gold answers, gpt-oss-20b writes the text and an independent Llama-3.1-8B-Instruct recovers every planned fact. An independent GPT-5.6 Sol check agreed with 85.4% of 48 sampled gold answers.
Install the jevbench package and point jb.systemone at any server implementing the interface, or wrap a Python function, then call jb.evaluate for JevBench-mini and read report.table. Saved reports load again with jb.Report.load.
Apache-2.0 for the data and reports, with no model weights included. The cases are synthetic and English, the gold verifier is a model, and the test set is public, so a model trained on it would no longer be measured fairly.
More Jev resources like this
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonjohannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.