Jev directory
CommunityJev Benchmarks & Evaluations
Aster-JEV-27B Results
Hugging Face dataset holding the scored public predictions, latency timings, and environment for the Aster-JEV-27B vision run: 9,867 of 12,210 requests correct, public index 76.7057, mean 507.2 ms per request on one A800 80GB.
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Self-scored public Vision index of 76.7057, with 9,867 of 12,210 requests correct and complete: true.
- Full public suite of 12,210 requests run on one A800 80GB at batch size 1.
- Inference mean 507.2 ms, median 361.5 ms, P80 581.9 ms; preprocessing and transfers excluded.
- Ships runs/Aster-JEV-27B scored predictions, runs/Aster-JEV-27B-latency-full timings, and reports/latency-full-summary.json.
- Official RTX PRO 6000 check (mean/median/P80 at or under 1,000 ms per request) is pending.
Questions about this entry
Short answers drawn from the linked project or page.
One row of JSON artifacts: complete scored public predictions under runs/Aster-JEV-27B/, vision-scores.json and environment, full-suite timed predictions under runs/Aster-JEV-27B-latency-full/, and reports/latency-full-summary.json with mean, median and P80. Complete request timings are retained separately from forward timings.
The self-scored public Vision index is 76.7057, with 9,867 of 12,210 requests correct and complete: true. The card states this is the public score and that private evaluation is performed by the maintainers.
The full public suite of 12,210 requests ran on an A800 80GB at batch size 1: inference mean 507.2 ms, median 361.5 ms, P80 581.9 ms. These are synchronized model forward times; preprocessing and transfers are excluded.
The dataset is published under CC BY-NC 4.0, and the fine-tuned weights are also CC BY-NC 4.0, while the original Qwen backbone is Apache-2.0. Benchmark questions and images are not redistributed in the dataset.
Download the dataset with hf download k2021211255/Aster-JEV-27B-Results --repo-type dataset, then run decision-index vision score --dir work/vision --results Aster-JEV-27B-Results/runs/Aster-JEV-27B/results.jsonl against the rebuilt suite.
Linux x86_64, Python 3.13, and one A800 80GB with an NVIDIA driver supporting CUDA 12.8. The whole model runs on cuda:0 with no distributed launcher or sharding, and the Winoground dataset terms must be accepted before rebuilding the suite.
It is a results artifact, not training data or model weights, and it does not include the benchmark questions or images. The RTX PRO 6000 check is still pending, so it supports no cross-hardware latency claim beyond the A800 numbers.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
A five-notebook Jupyter hands-on that runs Jev and Kev, an Apache-2.0 decision model served locally with llama.cpp, over the SMS Spam Collection, comparing accuracy and threshold effects across Kev-4B, 9B, and 27B.
Jev Benchmarks & Evaluations#evaluation#python#local
Communitydaikumatan/jev_kev_spam_handson
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
arXiv paper examining numerical nondeterminism in vLLM Split-K LoRA kernels and introducing a deterministic Triton Split-K implementation, measured on GSM8K, MBPP, and Jev-style structured toxicity decisions over Civil Comments.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Hugging Face dataset of Jev decision-index results from the mjev-vl-4b vision-language checkpoint, pairing decision and probability outputs with execution metadata, image budgets, latency, and SHA-256 hashes for weights and inference code.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Community
arXiv paper comparing two releases of Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks, to test whether an augmentation that presents labels in several surface forms helps when callers supply their own option wording.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.