CommunityJev Benchmarks & Evaluations
Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences (arXiv 2609.24965)
An evaluation of Jev as a semantic decision component in scientific workflows using a harness that assigns arithmetic to code. Across twelve model configurations, twenty source-grounded Choices, and ten scientific cases, Jev matched five configurations at complete semantic correctness.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Studies choices among known relations before a deterministic calculation can proceed.
- Compares twelve model configurations on twenty source-grounded Choices across ten scientific cases, each repeated five times.
- Measures semantic selections, downstream outputs, and final claim labels separately.
- Jev matched five other configurations at complete semantic correctness with the lowest median latency among successful responses.
- Seven wrong selections on one culture-history question changed downstream counts while preserving the correct final label.
Watch out
arXiv preprint posted September 21, 2026 and updated September 23, 2026 (cs.CL, cs.AI) by Boyuan Deng, Shuyi Fan, Hongyang Zhang, and Xinhong Xie; 10 pages, 1 figure, 5 tables. The cases are prepared source-grounded tasks, and the abstract states that evaluating Jev's role requires checking the relations and quantities a workflow will reuse.
Questions about this entry
Short answers drawn from the linked project or page.
Scientific workflows often require choosing among known relations before a deterministic calculation can proceed, such as whether observations share a culture, treatment, or reference standard.
A harness that follows Jev's documented guidance and assigns arithmetic to code, comparing twelve model configurations on twenty source-grounded Choices across ten scientific cases, each repeated five times.
Semantic selections, downstream outputs, and final claim labels are measured separately, so a correct final label cannot hide a wrong intermediate relation.
Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency among successful responses.
Across three comparison models, seven wrong selections on one culture-history question changed downstream counts while preserving the correct final label.
It identifies a useful role for Jev in prepared scientific decision tasks and shows why evaluating that role requires checking the relations and quantities that a workflow will reuse.
Teams wiring decision models into scientific pipelines, especially where a wrong relation silently changes the meaning of a count or comparison.
More Jev resources like this
A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
VictorYXL/GraphDecisionBenchPython
VictorYXL/GraphDecisionBenchPython0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An audit of 28 papers posted between 19 and 24 September 2026, nine days after TypeSafe released Jev, relating their findings to earlier work and deriving a 14-item evaluation checklist. The typed readout shows no independent accuracy advantage yet; the clearest gains are latency and cost.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A comparison of Jev with 19 LLMs across 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. Jev is competitive with frontier LLMs on knowledge and commonsense tasks, but on MathQA it trails the frontier median by 17.7 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.