Can I trust these third-party numbers at face value?

Read them as inspectable evidence, not settled results. The page links start-here audits with pre-registered probes, noise-floor comparisons, and machine-readable tables so you can recheck claims.

Which robustness effects are the largest in the table?

Criteria wording moves paired accuracy by tens of points, option names carry meaning rubrics do not override, and Noul versus two-option Choice on the same judgement differs by about 0.125 on average.

Does option order or batching actually change answers?

Reversing two options barely moves typical tasks but the first-listed option gains substantially on ambiguous questions, and arithmetic accuracy drops when the right answer moves last. Batching questions is safe while packing many rows is not.

What are the catches on injection, abstention, and language?

Injected text works when it reads as evidence about the judged item while blunt commands mostly fail, and without an explicit unknown option Jev answers anyway with high-confidence stereotype picks. Non-English inputs also lose accuracy and calibration.

What counts as robustness here, and what is left out?

An entry has to show how an answer or probability moves with question or option wording, option order or count, repeated calls, input language, planted steering text, or inputs that fit no option, or test whether the returned probability is calibrated. Tools, SDKs, apps and general directories are excluded, and accuracy-only task benchmarks live in a separate data file.

How many entries are there, and against which model version?

One hundred and forty-six entries, almost all run against jev-1.13.0, captured in a snapshot dated October 7, 2026. Most were done in the two weeks after the mid-September release, usually by one person on a small budget, so the collection is meant as evidence to inspect rather than settled results.

What is the largest single audit, and what did it cover?

The calibration audit: a 400-item comparison of Noul against Choice, negation complements, option order, batching with hostile neighbouring questions, Korean versus English, and the collapse that happens when an abstain option is available. It prints every calibration error next to its own sampling noise floor so numbers can be compared on one scale.

CommunityJev Guides & Articles

Awesome Jev Robustness

Curated collection of 139 independent Jev robustness tests on wording, order, repetition, language, injected text, and calibration, each grouped by the property measured.

See highlights

Open github.com

Stars: 6
License: CC0-1.0
Published by: Community
Author: Yifan-Lan
Last verified: Oct 2026

Yifan-Lan/awesome-jev-robustness

User repository on GitHub

View on GitHub

Tests, calibration audits and failure-mode studies of Jev (TypeSafe System One): jaggedness, consistency, injection, abstention.

Forks: 4
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

146 entries · almost all against jev-1.13.0 · snapshot 2026-10-07 · numbers are the authors' own. Most of these were run in the two weeks after Jev's release on 15 September 2026, usually by one person with a small budget. Read them as evidence to inspect, not settled results. Task benchmarks that only report an accuracy are kept out of this page; they are in data/taskbenchmarks.tsv. The machine-readable version of this page is data/entries.tsv.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Highlights

Groups 139 entries by measured property with per-property findings and linked sources.
Separates robustness probes from accuracy-only task benchmarks kept in a TSV file.
Highlights start-here audits covering calibration gaps, order effects, and injection cases.
Ships machine-readable entry and benchmark tables for independent comparison.

Watch out

Entries are the linked authors' own numbers, mostly small-budget studies, and accuracy-only task benchmarks live in a separate data file. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-hub

Static aggregation site and JSON dataset collecting Jev posts, demos, long-form pieces, and discussions from X with original links and author credit and no rehosted content.

Jev Guides & Articles#community#jevbest#catalog-import

Community

mizzlelover/jev-hubCSS

awesome-jev-usecases

Community index of shipped Jev decision-model projects across workflow, bulk, realtime, verify, harness and voice patterns, with official docs, SDKs and per-link rubric scoring.

Jev Guides & Articles#community#jevbest#catalog-import

Community

anandi1989/awesome-jev-usecases

awesome-jev-usecases

Community catalog of real Jev use cases, working architectures, and hard-won guidance, organized by decision shape with runnable examples and a ready-made question catalog.

Jev Guides & Articles#community#jevbest#catalog-import

Community

aliaihub/awesome-jev-usecases

awesome-jev-live

275

Self-rebuilding Jev index organized by evidence grade, with one ranked pick per category, multilingual guides, and links through to full cards and sources.

Jev Guides & Articles#community#jevbest#catalog-import

Community

wh000wh000/awesome-jev-livePython

awesome-jev

Curated link collection tracking Jev resources, open reproductions, integrations and architecture playbooks, with every entry resolved from its source thread and checked as a live repo.

Jev Guides & Articles#community#jevbest#catalog-import

Community

tanxarx/awesome-jev

awesome-jev

Curated index of Jev tools, integrations and experiments across frameworks, browser use, mobile, coding agents, MCP, observability, data, trading and games.

Jev Guides & Articles#community#jevbest#catalog-import

Community

ckaraca/awesome-jevPython

From the Jev guides

Original write-ups on what this entry is for.

Data Retention, Residency, and the Decision-Only Advantage

Oct 12, 2026

The typed-answer shape removes a category of data risk, not all of it. Zero data retention per surface, the legal set to read, and the controls that remain yours to run.

Jev Guides & Articles#privacy#compliance#legal

Read article

The Unofficial Jev Hosts and Playgrounds

Oct 11, 2026

TypeSafe serves Jev from one API. A second tier of independent sites wraps it with their own keys, credits, and playgrounds. How to read the disclaimers before you paste a key.

Jev Guides & Articles#ecosystem#trust#hosting

Read article

How Jev's Launch Landed

Oct 11, 2026

The first ten days produced three receptions that rarely meet: a Product Hunt launch, a 620-point Hacker News thread, and a r/LocalLLaMA that doubted the framing. Reading them together.

Jev Guides & Articles#launch#community#reception

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai