CommunityJev Benchmarks & Evaluations
JEVQA - Video Quality from Metadata, Bitstream, and Pixel Features with a General-Purpose Decision Model (arXiv 2609.24395)
A zero-shot video quality model that feeds encoding metadata, bitstream statistics, or pixel features to Jev. On 1,936 AV1, H.264, HEVC, and VP9 encodes, metadata-only reaches a Pearson correlation of 0.737 against VMAF, on par with ITU-T P.1204.1, and added features lift it to 0.824.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Uses Jev as a zero-shot video quality model returning distributions over a provided answer scale.
- Metadata-only JEVQA reaches a Pearson correlation of 0.737, on par with ITU-T P.1204.1 (0.733).
- Bitstream data raises accuracy to 0.797 and combined pixel and bitstream features to 0.824.
- A pixel-only variant failed in the authors' tests.
- In AVT-VQDB-UHD-1, metadata-only reaches 0.879 correlation with MOS, close to P.1204.1 (0.898).
Watch out
arXiv preprint posted September 21, 2026 (eess.IV) by Werner Robitza. Trained models on the same features remain clearly ahead in both studies, the pixel-only variant failed, and bitstream statistics did not help on AVT-VQDB-UHD-1, so the zero-shot correlations are promising rather than state of the art.
Questions about this entry
Short answers drawn from the linked project or page.
JEVQA is a zero-shot video quality method that uses Jev, a commercial System One model returning probability distributions over a provided answer scale, without task-specific training.
1,936 AV1, H.264, HEVC, and VP9 encodes of 22 sources, scored against VMAF as ground truth.
A Pearson correlation of 0.737, on par with the standardized ITU-T P.1204.1 model at 0.733.
Bitstream data raised accuracy to 0.797, and combined pixel-based and bitstream features raised it to 0.824; a pixel-only variant failed in the authors' tests.
On H.264, HEVC, and VP9 encodes in the AVT-VQDB-UHD-1 database, the metadata-only model reached a correlation of 0.879 with MOS, close to P.1204.1 (0.898), and bitstream statistics did not help there.
Yes. The authors report that trained models on the same features remain clearly ahead in both studies.
Usually every new input variable requires retraining a quality model; JEVQA tests whether a general-purpose model can answer quality questions zero-shot, which the authors call promising.
More Jev resources like this
An exploratory offline evaluation of decision readouts for text-mediated video anomaly detection on 40 videos and 400 anchors from UCF-Crime and XD-Violence. Jev Noul reaches 75.99% average precision on XD captions but shows no corresponding UCF advantage.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
4GitHub stars
An audit of whether Jev's confidence reflects missing knowledge rather than chance, using over 15 public datasets and 6 generated task families with paired interventions. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.