Docs
CommunityJev Benchmarks & Evaluations
JevBench Evaluation Cases
Cases and predictions from JevBench, an open evaluation framework for typed decision models: 24,799 decisions over public-dataset slices for hosted Jev, djev-0.1 and laya-421m-en.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- Leanmcp
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- 24,799 decisions across slices built from public datasets including banking77, mmlu_pro, medmcqa, sst5 and vqa_rad.
- Evaluates three systems: jev-1.13.0 hosted, djev-0.1 on a DiffusionGemma 26B-A4B backbone, and laya-421m-en on a ModernBERT-large encoder.
- Each cases/ row is the exact request sent to every model, plus the gold answer, source revision and licence.
- Predictions carry the full probability vector, latency and token counts; analysis/ holds paired bootstrap results and the djev image ablation.
Watch out
Dataset-level licence is 'other' because each case row carries the licence of its source dataset and licences do not merge across sources. SST-5 states no licence on its card, so its cases ship without the sentence text, leaving a state_hash recoverable from the pinned source revision. Predictions include no source text.
Questions about this entry
Short answers drawn from the linked project or page.
An open evaluation framework for typed decision models: it evaluates models that return a yes/no probability, a distribution over choices, or an expected level on an ordered rubric, on identical inputs built from public datasets.
Three: jev-1.13.0 as a hosted commercial model, djev-0.1 on a DiffusionGemma 26B-A4B backbone whose image slices live in djev-0.1-images, and laya-421m-en, a ModernBERT-large encoder.
24,799 decisions in total, split across cases/ JSONL slices such as banking77, jailbreak_classification, medqa_usmle, mmlu_pro, prompt_injections and scienceqa.
The harness that downloads sources, builds and verifies cases, runs models and scores them is released under the MIT Licence on GitHub at Leanmcp/jevbench; each predictions/<system>/scores.json holds the reported scores.
SST-5 states no licence on its card, so its cases are released without the sentence text; a state_hash identifies each sentence, which can be recovered from the pinned source revision recorded in sources.json.
Each case row carries the licence of its source dataset in a license field and licences do not merge across sources, which is why the repository-level tag is 'other'.
One decision per case with the full probability vector, latency and token counts, and no source text; djev-0.1 serving provenance includes the vLLM command line, container logs and configuration.
More Jev resources like this
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.