CommunityJev Benchmarks & Evaluations
Decision Readouts for Text-Mediated Video Anomaly Detection: An Exploratory Evaluation of Jev and Qwen (arXiv 2609.34180)
An exploratory offline evaluation of decision readouts for text-mediated video anomaly detection on 40 videos and 400 anchors from UCF-Crime and XD-Violence. Jev Noul reaches 75.99% average precision on XD captions but shows no corresponding UCF advantage.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates typed decisions and three Qwen readouts on 40 videos and 400 anchors from UCF-Crime and XD-Violence.
- On XD captions Jev Noul reaches 75.99% average precision versus 48.47% for Qwen generated probability.
- The paired Noul difference was 18.18 percentage points with a 95% bootstrap interval of 5.53 to 31.50.
- UCF shows no corresponding advantage: caption ROC-AUC is 52.26% for Noul versus 65.95% for ordinal likelihood.
- Jev Choice returned 776 valid responses out of 800 under the study's strict numerical policy.
Watch out
arXiv preprint posted September 28, 2026 (cs.AI, cs.CV); 19 pages, marked exploratory. The development sample is sparse (only 10 and 37 positives per dataset), the full-coverage comparison was blocked by interface failures, and the authors state the results establish neither a causal typed-interface benefit nor general superiority.
Questions about this entry
Short answers drawn from the linked project or page.
A sparse development sample of 40 videos and 400 target anchors from UCF-Crime and XD-Violence, each presented as a summary and ordered captions, with 20 source groups and 200 anchors per dataset.
Only 10 and 37 positives respectively, which the paper notes limits the analysis.
The original five-backend pilot requested 4,000 predictions; Jev Choice returned 776 valid responses out of 800 under the study's strict numerical policy, blocking its full-coverage quality comparison.
75.99% average precision, versus 48.47% for Qwen generated probability and 57.81% for the stronger local ordinal-likelihood expectation.
The paired difference was 18.18 percentage points with a 95% source-group bootstrap interval of 5.53 to 31.50.
No. Caption ROC-AUC was 52.26% for Noul and 65.95% for ordinal likelihood, and both probability readouts had worse UCF Brier scores than the evaluation-prevalence reference of 0.0475.
They describe the results as exploratory offline findings that establish neither a causal typed-interface benefit nor general superiority, calibration, or end-to-end acceleration, and note that a binary-likelihood control is missing.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: A one-token logprob wrapper that also classifies webcam frames.
allanrbo151 pointsA single function Jev-like wrapper for LLMs, including vision modelsRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
A zero-shot video quality model that feeds encoding metadata, bitstream statistics, or pixel features to Jev. On 1,936 AV1, H.264, HEVC, and VP9 encodes, metadata-only reaches a Pearson correlation of 0.737 against VMAF, on par with ITU-T P.1204.1, and added features lift it to 0.824.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark providing exact instance-level class probabilities across three simulated text domains: peer review, covert information transfer, and credit risk. Because the data-generating process is defined, exact probabilities and Bayes error are computable, enabling KL-divergence scoring of reported probabilities.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
4GitHub stars
An audit of whether Jev's confidence reflects missing knowledge rather than chance, using over 15 public datasets and 6 generated task families with paired interventions. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
RLCDAlignBench measures Jev as a zero-shot detector of ten alignment failures across 44 benchmarks and five target models. A single generic question reaches median AUROC 0.886 and beats supervised baselines on most benchmarks at 63 times lower cost than LLM judges.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
sumleo/RLCDAlignBenchPython
sumleo/RLCDAlignBenchPythonA study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
From the Jev community
Posts from builders shipping with Jev right now.