CommunityJev Benchmarks & Evaluations
Can Decision Models Understand Stance? Evaluating Jev Against General-Purpose LLMs (arXiv 2610.11901)
An evaluation of Jev on two stance detection datasets, VAST for English texts and ZS-CSD for Chinese conversations, against four general-purpose LLMs and two fine-tuned models. Jev matches GPT-5.6 on VAST but falls behind stronger LLMs on ZS-CSD.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates Jev on VAST (English texts) and ZS-CSD (Chinese conversations).
- Compares against four general-purpose LLMs and two fine-tuned models.
- Jev achieves competitive performance on VAST, matching GPT-5.6 and outperforming the other general-purpose LLMs.
- On ZS-CSD it falls behind stronger LLMs, particularly in distinguishing favor from against.
- The limitation appears related to understanding reply relationships and stance direction rather than conversation length alone.
Watch out
arXiv preprint posted October 8, 2026 (cs.CL; 8 pages) by Xing Li, Jinzhong Ning, Yijia Zhang, Liang Yang, and Hongfei Lin. The abstract reports dataset-level results without confidence intervals or a code release, and the failure analysis on reply relationships is suggestive rather than conclusive.
Questions about this entry
Short answers drawn from the linked project or page.
Identifying an author's attitude toward a given target, sometimes based on conversational context, which general-purpose LLMs are commonly used for.
VAST, containing English texts, and ZS-CSD, containing Chinese conversations.
It achieves competitive performance, matching GPT-5.6 and outperforming the other general-purpose LLMs in the comparison.
On ZS-CSD it falls behind stronger LLMs, particularly in distinguishing favor from against.
Further analysis suggests it may relate to understanding reply relationships and stance direction rather than conversation length alone.
Four general-purpose LLMs and two fine-tuned models, alongside Jev on both datasets.
It highlights both the potential and limitations of Jev, with competitive English performance but weaker results on the Chinese conversational set.
More Jev resources like this
A pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HATEDECIDE evaluates six decision-model configurations on four hate-speech datasets against specialized, zero-shot, commercial, and supervised baselines. The best hosted model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A first empirical study of Jev on application classification for network flows, using ten packet-header features from CESNET-QUICEXT-25. Forty labeled examples raise accuracy from 9.80% to 28.42%, while tree ensembles trained on 8,000 records reach about 67-70%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
VictorYXL/GraphDecisionBenchPython
VictorYXL/GraphDecisionBenchPythonFrom the Jev community
Posts from builders shipping with Jev right now.