User repository on GitHub
Benchmarking Jev’s Capabilities on Graph Tasks
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
GraphDecisionBench: Benchmarking System One Models on Graph Tasks (arXiv 2610.06354)
A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.
- Stars
- 6
- Language
- Python
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- Combines structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls.
- Evaluates Jev and related choice-based models alongside language-model baselines across fourteen configurations.
- Accurate adjacency recognition does not guarantee broader structural correctness in Jev's results.
- Joint graph-text input does not consistently improve prediction, and feasible construction does not establish high solution quality.
- Code and aggregate results are available at github.com/VictorYXL/GraphDecisionBench.
Watch out
arXiv preprint posted October 5, 2026 and updated October 12, 2026 (cs.AI) by Xianliang Yang, Yapu Zhang, and Li Zhao. The abstract reports diagnostic distinctions rather than a full accuracy table, and the fourteen model-interface configurations are not described individually.
Questions about this entry
Short answers drawn from the linked project or page.
Graph decision performance, using structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls that make the benchmark model-independent.
Jev and related choice-based models alongside language-model baselines, covering fourteen model-interface configurations.
Accurate adjacency recognition does not guarantee broader structural correctness, according to the benchmark's central distinctions.
Not consistently. Joint graph-text input does not consistently improve prediction in the reported results.
No. Feasible construction does not establish high solution quality, one of the distinctions the benchmark is built to surface.
Code and aggregate results are available at github.com/VictorYXL/GraphDecisionBench.
Its task contracts, candidate interfaces, and scoring rules support comparison across native selectors and language-model adapters.
More Jev resources like this
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A comparison of Jev with 19 LLMs across 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. Jev is competitive with frontier LLMs on knowledge and commonsense tasks, but on MathQA it trails the frontier median by 17.7 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
7GitHub stars
A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonA benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonFrom the Jev community
Posts from builders shipping with Jev right now.