Article
CommunityJev Benchmarks & Evaluations
Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation
Independent audit of a frozen System One model on 142,658 human-labeled comments, benchmarked against 14 frontier reasoning models. It reports AUROC 0.83, ~USD 0.000025 per call, ~100 ms, and separates human disagreement from decision risk.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Audits one frozen System One model on 142,658 human-labeled comments.
- Benchmarks against 14 frontier reasoning models on a shared probe set.
- Reports zero-shot classification AUROC of 0.83 for the cheap model.
- Costs about USD 0.000025 per call and runs at roughly 100 ms.
- Separates human disagreement from decision risk under asymmetric moderation loss.
Questions about this entry
Short answers drawn from the linked project or page.
It runs an independent audit of one frozen System One model on 142,658 human-labeled comments and benchmarks it against 14 frontier reasoning models on a shared probe set.
Its zero-shot classification AUROC is 0.83, which the paper says is not statistically separable from 13 of the 14 frontier systems under a paired bootstrap with n=56.
The paper reports about USD 0.000025 per call and roughly 100 ms latency, which it contrasts with running a frontier LLM on every post.
It separates human disagreement, where annotators split on satire and coded language, from decision risk, where the automatic action is costly-wrong under an asymmetric moderation loss.
No. The paper argues the two uncertainties call for opposite escalation signals, so no single confidence number serves both.
Because extra questions are priced at nearly zero, it decomposes the judgment into parallel sub-questions and reads their spread, which recovers a signal the model's native confidence misses.
It helps the weakest, disagreement-blind models and only when the decomposition aligns with the sources of disagreement; a specialist toxicity model and the strongest LLMs do not need it, and the effect is distribution-sensitive.
More Jev resources like this
arXiv paper comparing two releases of Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks, to test whether an augmentation that presents labels in several surface forms helps when callers supply their own option wording.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Retrospective study comparing JEV, a non-generative decision model returning native option probabilities, with schema-constrained Qwen Flash on 2,048 patient-condition pairs from TrialGPT and MIMIC-III/IV under Zero Data Retention policies.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
arXiv paper examining numerical nondeterminism in vLLM Split-K LoRA kernels and introducing a deterministic Triton Split-K implementation, measured on GSM8K, MBPP, and Jev-style structured toxicity decisions over Civil Comments.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.