Why was this benchmark published?

The README says most numbers circulating about Jev are either the vendor's own or arithmetic from the rate card; these are what the live API actually returned, with raw data in data/measured.json and reproducibility via your own key.

How much does one Jev decision cost in the measurements?

Between $0.0000153 and $0.0000304 across twelve real use cases, with 364-724 input tokens at $0.042 per 1M and no output tokens billed — 13x to 26x below the roughly $0.0004 figure quoted elsewhere.

How fast is Jev in these measurements?

458-563 ms raw from a laptop, of which about 199 ms is the network floor, and a 184 ms median from a US-East server through OpenRouter. The README also notes that five questions in one request take the same time as one.

Is Jev more accurate than chat models?

It ties, it does not win: 27/27 labelled tickets, matching mistral-small-3.2 at 1.4x the price and 4x slower, while gemini-2.5-flash-lite and gpt-5-nano scored 26/27. Only Jev also returned a confidence: 0.979 clear, 0.841 ambiguous.

Which decision model was most accurate across 633 items?

d1 at 96.5% and Decider V1.1 at 96.0% on simple choices; with 59-77 options, Clef Flash scored 92.0% and Clef 89.0%, against Jev at 94.0% and 79.0% and GPT-6 Luna Decisions at 94.1% and 79.8%.

What is surprising about Score and Noul outputs?

Score returns the probability-weighted mean of the rubric indices (0 to n-1), not a 0-1 value, and Noul has no confidence field. Both are implementation details that can surprise code written against the wrong assumptions.

Why does typesafe/jev fail on OpenRouter chat completions?

Because Jev is a decisions model, not a chat model: the README's gateway gotcha says to use POST /api/alpha/decisions rather than /chat/completions. The repository documents the error for gateway users.

Jev directory

CommunityJev Benchmarks & Evaluations

Jev Measured

Measured cost, latency, accuracy and raw output from the live Jev API across twelve realistic use cases, plus twelve other decision models on the same 633 items. Reproducible with your own key; the whole measured Jev run cost under one cent.

See highlights Open github.com

Stars: 1
Language: Python
License: MIT
Reading time: 2 min
Published by: Community
Author: wallerchen
Last verified: Oct 2026

wallerchen/jev-measured

User repository on GitHub

View on GitHub

Measured cost, latency, accuracy and raw output: TypeSafe's Jev and 12 other decision models (OpenAI Decisions API, Clef, Decider, d1…) — reproducible

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity catalog-import cobanov benchmark cost

Works forJev for Evaluation

Used inJev for Software & DevTools

Languages: Python
Platforms: OpenRouter
Capabilities: Open sourceRequires API key

Highlights

One decision costs $0.0000153 to $0.0000304 across twelve use cases, 13x to 26x below the ~$0.0004 figure in circulation.
Raw latency 458-563 ms from a laptop with a ~199 ms network floor; 184 ms median from a US-East server through OpenRouter.
Accuracy ties rather than wins: 27/27 labelled tickets, the same as mistral-small-3.2; gemini-2.5-flash-lite and gpt-5-nano scored 26/27.
Score returns the probability-weighted mean of rubric indices (0..n-1), not 0-1, and Noul has no confidence field at all.
Five questions in one request return as fast as one: each question is billed as input tokens, but the state is billed once per request.

Watch out

Copy reviewed against the repo README on 2026-10-15; the measurements are dated (cost and latency on 2026-09-18 with four more use cases on 2026-10-11, accuracy on 2026-09-20, rival models on 2026-10-08) and the README is truncated in the digest after its findings. Raw data lives in data/measured.json.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: 100,000 viral posts in 20.4 seconds

X: SEO and GEO fixes, 90% cheaper

X: Hard-coded rules moved to Jev

YouTube: Joe Maddalone tries Jev as a routing and filtering step, with a transcript-analyzer demo.

From the Jev guides

Original write-ups on what this entry is for.

The trust ledger: measuring confidence your domain has earned

Oct 16, 2026

ASSAY-001 calibrated Jev on CLINC150 but found it overconfident on Banking77 — one audit, two verdicts. How calibration research measures the model-and-domain pair, discounts stated confidence, and signs the evidence.

Jev Benchmarks & Evaluations#calibration#evaluation#open-models

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai