What is this repository for?

It is a stable framework for running Jev tasks at scale with reproducible results, plus the experiment reports themselves: lightweight examples for getting started, full experiments with analysis, and a catalog of Jev-related projects.

How well did Jev route requests to agent skills?

Across a catalog of 126 real agent skills and 500 prompts, 250 tasks in a formal and a casual wording, Jev answered 488 correctly, 97.6%, including tasks that required choosing from supplied skill combinations or deciding that no skill was needed.

What did the graduate science benchmark show?

On 198 biology, physics and chemistry questions from GPQA Diamond, each with four answer options, Jev answered 150 correctly for 75.76%; the report ships with the prompts and scoring so the run can be repeated.

Where did the housing-price experiment struggle?

Comparing two suburbs was strong at 88.2% correct, but converting a single suburb's score into a price range was right only 9.7% of the time, and choosing the range it considered most likely reached 31.4%.

What kinds of experiments are included?

Benchmarks such as GPQA Diamond, GSM8K, MMLU-Pro, BBQ and SocialIQA; routing and classification studies including skill routing, prompt routing and paper classification; numeric prediction on Boston Housing and Titanic; and DPO and GRPO annotation tests.

How do I run or add an experiment?

The framework lives under src/decision_models with schemas for datasets and results, and run.py is the CLI entry point; the README recommends starting from the lightweight examples under example/ before the full experiments.

Does one number grade Jev's general capability?

No, each figure describes one dataset and prompt set. The README presents the results as findings from this framework, including negative ones such as the housing regression, rather than a single benchmark score.

CommunityJev Benchmarks & Evaluations

What the Jev

A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.

See highlights

Open github.com

Stars: 9
Language: Python
License: MIT
Published by: Community
Author: keta1930
Last verified: Oct 2026

keta1930/what-the-jev

User repository on GitHub

View on GitHub

What can Jev actually do? Reproducible experiments and research reports exploring its capabilities and limits.

Forks: 1
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

Tagscommunity python evals benchmark research

Works forJev for Evaluation Jev for Routing Jev for Classification

Used inJev for Software & DevTools Jev for Data & Analytics

Languages: Python
Platforms: CLI
Capabilities: Open source

Highlights

Skill routing: with 126 real agent skills and 500 prompts, Jev chose correctly 488 times (97.6%), including deciding no skill fit.
On 198 GPQA Diamond graduate science questions with four answer options each, Jev answered 150 correctly (75.76%).
Boston Housing: Jev picked the pricier of two suburbs 88.2% of the time, but a single suburb's price range was right only 9.7%.
Covers lightweight examples such as election recall, ticket triage and prompt-injection detection plus full experiments from GSM8K to DPO annotation.
Ships a run.py CLI, dataset and result schemas, a bilingual documentation site, and a catalog of related models, tools and integrations.

Watch out

MIT. It is an eval workspace rather than a library: datasets, schemas and a CLI live here, and every experiment calls a hosted model, so the reported numbers belong to the prompts and model version used at the time of the run. The README publishes weak results as well as strong ones.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

Autoresearch

More Jev resources like this

Zero-Shot IE Bench

A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.

Jev Benchmarks & Evaluations#community#python#benchmark

Community

umstek/zero-shot-ie-benchPython

System One on Snapdragon

A dated harness for running open System One typed-decision models — Laya, Decider, imajev, Winnow-12B, decider-12b — on a Snapdragon X Elite NPU, GPU, and CPU, benchmarked against TypeSafe's Jev API.

Jev Benchmarks & Evaluations#community#python#snapdragon

Community

esterhuizen/system-one-on-snapdragonPython

GraphDecisionBench: Benchmarking System One Models on Graph Tasks (arXiv 2610.06354)

A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

VictorYXL/GraphDecisionBenchPython

Benchmarking candidate coverage and rejection policy transfer in typed decision models (arXiv 2610.03387)

A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

luckykevvv/Decision_Model_BenchmarkPython

Evaluating and Benchmarking the System One Model Jev (arXiv 2609.37647)

A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

AppliedMachineLearning-Lab/jev-benchmarkingPython

Open-Jev v1.1 Data

326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai