CommunityJev Benchmarks & Evaluations
To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation (arXiv 2610.03324)
HATEDECIDE evaluates six decision-model configurations on four hate-speech datasets against specialized, zero-shot, commercial, and supervised baselines. The best hosted model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates six decision-model configurations on four hate-speech datasets.
- Commercial LLMs significantly outperform all decision models on only one of the four datasets.
- Supplying a dataset's definition changes up to 28% of predictions without consistently improving classification.
- Question decomposition significantly improves performance in only 20% of the comparisons.
- The best hosted decision model is within 1.6 macro-F1 points of the best commercial LLM at roughly 97% lower inference cost.
Watch out
arXiv preprint posted October 2, 2026 (cs.CL) by Demetris Paschalides, George Pallis, and Marios D. Dikaiakos. The diagnostic set supporting the 1.6-point gap is small relative to the four datasets, and the abstract does not name the hosted model or the baselines individually.
Questions about this entry
Short answers drawn from the linked project or page.
Six decision-model configurations on four hate-speech datasets, against specialized moderation, zero-shot, commercial, and supervised baselines.
Commercial LLMs significantly outperform all decision models on only one of the four datasets, according to the abstract.
It changes up to 28% of predictions without consistently improving classification.
Decomposition significantly improves performance in only 20% of the comparisons.
On a diagnostic set of test cases, the best hosted decision model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost.
They identify opportunities for inexpensive moderation, while showing that explicit criteria and additional questions do not reliably improve classification.
No. The evaluation concerns structured decision models that accept natural-language criteria and select among specified answers without task-specific training.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Building a contextual brand-safety scorer on Jev from scratch.
X: A website slop detector that checks 35 tells in 243 ms
X: 100,000 viral posts in 20.4 seconds
X: Breaking, golden nugget, or slop
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
VictorYXL/GraphDecisionBenchPython
VictorYXL/GraphDecisionBenchPython0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A comparison of Jev with 19 LLMs across 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. Jev is competitive with frontier LLMs on knowledge and commonsense tasks, but on MathQA it trails the frontier median by 17.7 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of Jev on two stance detection datasets, VAST for English texts and ZS-CSD for Chinese conversations, against four general-purpose LLMs and two fine-tuned models. Jev matches GPT-5.6 on VAST but falls behind stronger LLMs on ZS-CSD.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev community
Posts from builders shipping with Jev right now.