Are the benchmark numbers real?

Per the digest, a 113-case first round plus margin, 100-question, and hard suites with raw logs: counting 20/20, dates 20/20, decimals 10/10, facts 9/10, injection 5/5, routing 10/10 per language, arithmetic near-threshold weak.

Does low confidence at least flag the wrong answers?

Only partly. Wrong arithmetic averaged 0.64 with most above 0.6 and two above 0.8, and one forced-choice misroute hit 0.87, so gate high-stakes calls on thresholds plus code checks rather than confidence alone.

Should I let Jev do the arithmetic inside the prompt?

No. The digest concludes it estimates magnitude but not exact multiply-add, so hand computed totals to the model and let it compare, which scored perfectly across the whole margin gradient.

What breaks with long lists or ambiguous dates?

Counts of 20 to 40 items dropped to 23/30, off by one or two at low confidence, and ambiguous numeric formats were answered one way at up to 0.98 without flagging ambiguity, so normalize inputs first.

How do I run it, and what does a full pass cost?

It needs only the Python standard library. Write TYPESAFE_API_KEY into .env, then run python3 run.py for the full suite, or add --limit 1 for a smoke test and --parallel for the latency test. A snapshot on 2026-09-19 sent 40,501 input tokens for about $0.0017, at the listed $0.042 per million input with output free.

Does it test Chinese and English separately?

Yes. Arithmetic, counting, dates, and routing each run Chinese and English sets side by side, because TypeSafe documents English as the primary training language. In the first round English arithmetic answered over 43 times out of 50, while routing scored 10/10 in each language.

How does knowledge freshness affect the factual tests?

It shows up as a single miss. The 9/10 fact round failed only on whether TypeSafe released Jev in September 2026, which the model rated true at probability 0.16, below its knowledge cutoff. Every other answer is program-computed or agreed fact, so this is the one fact question tied to the model's training data.

CommunityJev Tools & CLIs

Jev Reality Check

Independent Chinese-language test harness probing Jev `jev-1.13.0` where vendor claims hold and break: program-computed ground truths, recorded confidence and end-to-end latency, stdlib-only Python reruns.

See highlights

Open github.com

Stars: 1
Language: Python
License: MIT
Published by: Community
Author: ruffood
Last verified: Oct 2026

ruffood/jev-reality-check

User repository on GitHub

View on GitHub

Jev (TypeSafe AI) 可复现实测：算术、计数、日期、零幻觉、把握度校准，中英对照

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity github

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Covers arithmetic, counting, dates, decimal compare, facts, forced choice, injection, routing, and parallel fan-out.
Finds exact multiply-add weak but code-supplied totals perfect: 58/100 self-computed versus 100/100 assisted.
Shows parallel 1/5/20-question medians near 740/621/632ms with full reruns costing under a cent.
Runs on stdlib-only Python with a fixed seed; latency includes local network unlike server-side metrics.

Watch out

Latency figures include local network and outputs may shift with model versions; English is the primary training language. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-deep-dive

Long-form Jev writeup that maps each conclusion to official docs, vendor evals, integrations, community reproductions, and press, separating contract facts from unverified claims.