Are the numbers on the board real, and can I check them myself?

The README's claim is that every number can be recomputed from the logs in the repository. releases/ /board.json is the published board the site renders, runs/ __ __ .jsonl holds one decision per line with its typed output, usage, cost, malformed and refusal flags, and the formulas for Decision Score, accuracy, ECE, confidence intervals, ranks, the confidence gate, flip rates, cost and latency are documented on the methodology page.

Why can a Decision Score be negative?

The scale is anchored on the label base rates: 100 is perfect, 0 is what you get by guessing the majority label, and below 0 is worse than guessing outright. Two models land under it on HelpSteer2 helpfulness, which is the clearest signal in this data that scored rubrics on open-ended helpfulness are hard for every model here, Jev included.

Does this show Jev is better than the general-purpose LLMs?

It shows a cost and speed profile more than a quality win. Jev costs $0.029 to $0.043 per 1,000 decisions with p95 latency around 653 to 693 ms across the three tasks, which is where the value argument sits. On quality it leads or ties on the two closed-label tasks and ties for rank 1 of 8 on HelpSteer2, where nothing clearly beats guessing, so read the per-task boards rather than the summary line.

Can I use this data to run my own evaluation?

Yes, subject to licensing. The repository is CC BY 4.0 and asks for citation as Jevals with the release, suite and licence, but the underlying PubMedQA, Banking77 and HelpSteer2 datasets keep their own licences, listed in the NOTICE file. Suites ship the source dataset name and revision, the options, the instructions and each item's id, upstream row index and label, which is enough to rebuild the exact requests.

How are choice options ordered across the five epochs?

Epochs 0 and 1 replay order_seed 0 on identical requests to measure the repeat flip rate, while epochs 2 to 4 use seeds 1 to 3; only choice options are shuffled, since noul and score keep the task file's order.

Is the item text included with the jevals-data logs?

No. Item text is deliberately absent; each item points at its row index in the public source dataset together with the dataset name and revision, so reproducing a run means fetching the upstream data at that revision.

How much of the jevals benchmark is published?

One release is published so far, 2026-09-18 suite 0.1.0 with eight systems, and the repository asks you to open an issue with the release and row or item id when a number or label is wrong.

CommunityJev Practices & Patterns

Jevals Data

Ground-truth benchmark data for Jev and six LLMs answering the same typed questions on PubMedQA, Banking77 and HelpSteer2, published as a board plus per-decision JSONL logs so every number can be recomputed.

See highlights

Open github.com

Stars: 1
License: CC-BY-4.0
Published by: Community
Author: Jevals
Last verified: Oct 2026

Jevals/jevals-data

User repository on GitHub

View on GitHub

Independent benchmark data for TypeSafe's Jev (System One model) vs LLMs: accuracy, calibration, cost. Boards + per-decision logs, CC-BY-4.0

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

The data behind jevals.com, an independent, ground-truth benchmark of TypeSafe AI's Jev (the first System One model, called as typesafe-ai/jev through Vercel AI Gateway) and the LLMs that answer the same typed questions: noul (yes or no), choice (pick one of N) and score (place the state on a rubric). Every answer carries probabilities and is scored against human labels for Decision Score, accuracy, calibration, cost and speed. New to Jev? What is Jev? The System One model, explained.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Highlights

Decision Score is normalised so 0 means guessing the label base rates and anything below 0 is worse than that.
Each run log records item id, epoch, target label, option order seed, typed output, usage, cost and retries per decision.
Item text is deliberately absent; every item points at its row in the public source dataset together with its revision.
Choice options are shuffled deterministically from an FNV-1a hash of the item id and order seed, and the code is included.

Watch out

The headline result is weaker than the table looks. On HelpSteer2 helpfulness no model clearly beats guessing, and Jev's 9.2 sits barely above the 0.0 label prior while DeepSeek V4.1 Flash at -19.0 and Mistral Medium 3.5 at -13.7 fall below it. The prose summary also calls the PubMedQA noul result a tie at 1/28 of the price while the board table lists Gemini 3.8 Flash ahead at 73.0 against Jev's 69.0, so read the table rather than the headline. Only one release is published, covering eight systems in one suite. Repeated-decision structure is asymmetric: epochs 0 and 1 replay order_seed 0 to measure the repeat flip rate while epochs 2 to 4 use seeds 1 to 3, and only choice questions have their options shuffled at all, since noul and score keep the task file's order. Item text is not vendored, so reproducing a run means fetching the upstream datasets at the recorded revisions, which keep their own licences under this repository's CC BY 4.0. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.