Docs
CommunityJev Benchmarks & Evaluations
Visual Jev Decisions v1
A visual grounding dataset for decision training: 694,255 train, 24,137 validation and 6,254 test records over COCO 2017 boxes and Visual Genome attributes, with five task types and images downloaded separately.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- harshnandwana
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- 694,255 training, 24,137 validation and 6,254 test records; 6,823 gpt-6-luna candidates sit outside all splits pending human audit.
- Five task types: ground_bbox, box_choice, spatial_boolean, attribute_text and relation_text, with boxes normalized [x1,y1,x2,y2].
- Geometry rows derive from COCO 2017 instance boxes; Visual Genome v1.2 supplies color attributes and selected relations.
- Images are not bundled because COCO says image rights belong to their owners; images.jsonl records upstream URLs and source licences.
- One held-out photo yields 61 records: 8 grounding, 8 box choice, 38 spatial, 4 color and 3 relation questions.
Quickstart
bash
python3 dataset.py download-images train.jsonl --workers 12
python3 dataset.py download-images validation.jsonl --workers 8
python3 dataset.py download-images test.jsonl --workers 8
python3 dataset.py validate train.jsonl --check-imagesWatch out
Card reviewed 2026-10-17; the card says this is a research dataset for visual grounding and decision training, not a general visual benchmark, that annotation-derived labels are not individually reviewed ground truth and Visual Genome labels are noisy, and that it holds no reliable absence, count, OCR or calibrated UNKNOWN examples. Licence is 'other' and image rights stay with owners.
Questions about this entry
Short answers drawn from the linked project or page.
A research dataset for visual grounding and decision training with 694,255 training, 24,137 validation and 6,254 test records, built from COCO 2017 instance boxes and Visual Genome v1.2 attributes and relations.
Five: ground_bbox for normalized grounding boxes, box_choice with two distractor boxes, spatial_boolean left/right decisions, attribute_text color questions and relation_text questions drawn from Visual Genome.
COCO says image rights belong to their individual owners, so the repo ships images.jsonl with each photo's upstream URL, Flickr URL and source licence; download referenced images with the dataset.py download-images helper.
Geometry rows derive from COCO 2017 instance boxes; the visual_genome_annotation_pending_audit rows use Visual Genome v1.2 color attributes and selected relations for COCO-overlapping images, and the card warns those labels are noisy.
6,823 gpt-6-luna candidate records in luna_candidates.jsonl are excluded from training and evaluation pending human audit because they were generated and checked by the same model family.
No reliable object-absence, exhaustive-count, OCR or calibrated UNKNOWN examples, and published rows largely use fixed question templates, so unfamiliar wording, counting and false-premise questions are not validated.
Give the model the image, question, choices and, where present, candidate boxes; never include answer_index, answer_box_xyxy, answer_text, evidence, quality_tier or provenance in the prompt, and use answer fields only as targets.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
More Jev resources like this
Chinese-language research project for multimodal social-risk moderation: a local Qwen3-VL-32B with language-side LoRA emits structured evidence JSON, and a fixed Jev policy turns that evidence into risk bands, with frozen test leaderboards and pure-Jev ablations.
Jev Benchmarks & Evaluations#community#moderation#multimodal
A System One model family for video: VideoJev-4B and VideoJev-35B-A3B perform temporal grounding and multiple-choice video QA by scoring candidates directly, returning probabilities instead of generated text.
Jev Models & Reimplementations#community#catalog-import#cobanov
Community
mcg-nju/videojevPython
mcg-nju/videojevPythonA worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonA 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.