Docs
CommunityJev Benchmarks & Evaluations
JEV-Contrast
A multiple-choice visual QA dataset for candidate-dependent evidence, published under the tudo-vn account: each item has one image, one question and two candidate sets of 3 to 10 options, each set needing a different evidence region. For research and development only; annotations are CC BY 4.0.
- Format
- Docs
- Reading time
- 3 min
- Published by
- Community
- Author
- Tudo VN
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- Each item offers two candidate sets; a set has 3 to 10 options, the maximum of the Jev Decision Index vision board.
- 29,225 train records, 430 validation and 1,951 test, stored as parquet with embedded images, about 4.2 GB.
- Every wrong option differs from the correct answer in one detail, so each set needs a different image region checked.
- Evidence regions per set are annotated as label, role (answer or option) and box_xyxy.
- Version 0.4.1, created 2026-10-16; validation and test wrong options passed a vision-language model check.
Quickstart
python
from datasets import load_dataset
ds = load_dataset("nvidia/JEV-Contrast")
ex = ds["test"][0]
ex["set1_options"], ex["set1_answer_index"]
ex["set1_evidence"]Watch out
Card reviewed 2026-10-17; the annotations are CC BY 4.0 but images and source annotations keep their original terms (Visual Genome and GQA CC BY 4.0, COCO under Flickr per-image terms, LVIS v1 CC BY 4.0, VAW per its repository), and the card limits the dataset to research and development. The card's usage example loads an nvidia/JEV-Contrast repository, not this account's copy.
Questions about this entry
Short answers drawn from the linked project or page.
A multiple-choice visual question answering dataset where each item has one image, one question and one correct answer, plus two candidate sets. Every option in a set differs from the correct answer in one detail, so each set requires checking a different evidence region of the image.
Between 3 and 10, the maximum of the Jev Decision Index vision board. An n = 3 item has the same options as in v0.3.x, and validation and test balance option counts so the values do not pile up at the maximum.
Train holds 29,225 records, validation 430 and test 1,951, stored as parquet with embedded images plus light JSONL annotations without images, about 4.2 GB in total. Version 0.4.1 was created on 2026-10-16.
Because option counts differ, the card says to report chance-corrected accuracy per item, (acc - 1/n) / (1 - 1/n), averaged over items, the same way the Jev Decision Index does.
In train, the set option columns hold all options available for a set (up to 10, shuffled). Sample the number of options n and which wrong options to keep at training time, always keep the correct one, and mask the answer distribution to the shown letters.
Each set carries evidence regions as label, role (answer or option) and box_xyxy in the image's coordinate frame. Validation and test wrong options were judged clearly contradicted by a vision-language model; train used a faster, less strict check.
The annotations are CC BY 4.0, but images and source annotations keep their original terms, including Flickr per-image terms for COCO. The card states the dataset is for research and development only, and users are responsible for compliance.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Happy JevDay: images, box, longer context, and an opt-in to find jaggedness
X: First among models under 10B on the Decision Index v0.2.1. Liquid AI's d1-3B is built from LFM2.5-VL-3B and…
X: The classic hotdog classifier as a real-time vision decision. Fayaz Ahmed rebuilt the not hotdog app on…
X: Liquid d1 with vision
More Jev resources like this
A worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonMetamorphic coherence benchmark for typed probabilistic decision models: 240 synthetic cases and 2,240 questions checked by 50 probability laws, with released reports for 21 open models and 4 reference models.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
0GitHub stars
Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScriptAn open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.