Docs
CommunityJev Benchmarks & Evaluations
JEVal — huggingface.co
A benchmark of general decision models that unifies 36 datasets, 58 subtasks and 11,257 examples in one test split, with choice, noul and score questions in English and Chinese and 25 configurations scored.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- carlosxiang
- Last verified
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is Jeval — rlaope.
- Languages
- Python
Highlights
- 11,257 examples from 36 datasets and 58 subtasks across 10 domains: 9,582 choice, 1,356 noul and 319 score questions.
- 9,457 English and 1,800 Chinese examples in one Parquet file and a single test split, with source and question-type filters.
- Paper Table 1 scores 25 configurations: Jev 78.38% accuracy with ECE 0.0532, InnerJev-27B 79.24 with 0.0149, GPT-6-Luna 80.93 with 0.1159.
- Releases InnerJev-4B and InnerJev-27B, based on Qwen3.5-4B and Qwen3.8-27B, as full-rank checkpoints plus LoRA adapters.
- First-Token Decision Readout reads option-label logits in one forward pass; self-distillation trains it to match post-reasoning answers.
Quickstart
python
import json
from datasets import load_dataset
dataset = load_dataset("carlosxiang/JEVal", split="test")
row = dataset[0]
native = {
"source": row["source"],
"state": row["state"],
"questions": json.loads(row["questions_json"]),
"label": json.loads(row["label_json"]),
}Watch out
Card reviewed 2026-10-17; the card states no licence for the dataset or the InnerJev checkpoints, accuracy counts invalid responses as incorrect, and all headline numbers are the authors' own evaluation in arXiv:2610.03935 rather than independent reproductions. Score questions are evaluated separately from the choice and noul set.
Questions about this entry
Short answers drawn from the linked project or page.
A benchmark for general decision models — models that return structured decisions and probabilities over candidate options — unifying 36 datasets, 58 subtasks and 11,257 evaluation examples across 10 application domains in one test split.
It benchmarks decision models in the typed interface shape; Jev is one of the 25 configurations scored, at 78.38% accuracy with 0.0532 ECE across the 10,938 choice and noul questions, second to GPT-6-Luna's 80.93% on accuracy.
Three: 9,582 choice questions over listed options, 1,356 noul yes/no questions and 319 score questions on ordered levels; only state and questions enter a model request, and labels are used for scoring.
InnerJev-4B on Qwen3.5-4B and InnerJev-27B on Qwen3.8-27B, each as a full-rank checkpoint and a LoRA adapter, with training data released separately; InnerJev-27B scores 79.24% accuracy with 0.0149 ECE in the paper's table.
load_dataset("carlosxiang/JEVal", split="test") or the Parquet path; rows carry source, state, questions_json and label_json, and only state and questions should enter a request while the label is for scoring.
No licence appears on the card or in its metadata; the dataset is tagged with an arXiv identifier and benchmark tags only, so check the paper and repository before reuse.
They are the authors' own evaluation: accuracy counts invalid responses as incorrect, ECE and raw Brier use valid responses, the 319 score questions are evaluated separately, and no independent reproduction is listed.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Both Jev 1.13 and 1.14 score 29 of 30 on 30 Turkish buyer and seller messages
More Jev resources like this
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPythonA 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.