How does it get a perfect parse rate on thousands of items?

By construction: the four options are the response schema, so Jev cannot emit an out-of-schema letter or bury the answer in prose, and every record still carries per-option probabilities for calibration.

Is the headline total just pooled per-question accuracy?

It averages each subject within its broad category, then averages the four category means with equal weight, and reports pooled micro accuracy separately so large categories do not dominate.

Should I run jev-latest or jev-preview for a repeat sweep?

The repo reports the two aliases resolving to the same backend version with cross-alias agreement near same-alias rerun agreement, and recommends jev-latest for its steadier tail latency.

What does a full 20k-call run cost me if it gets interrupted?

Install the editable package, export TYPESAFE_API_KEY, smoke-test a few subjects with a per-subject limit, then run the full resumable sweep with concurrency; interrupted runs pick up where they stopped.

What benchmark and model does this harness evaluate?

TMMLU+ v1.1, the 66-subject Traditional Chinese benchmark; every question becomes one four-way choice (A/B/C/D) to jev-latest, covering 19,646 questions across 66 subjects with 0 API errors.

How is Jev's TMMLU+ score calculated?

The headline Total follows the ievals methodology: average each subject's accuracy within its broad category, then average the four category means with equal weight; pooled micro accuracy is reported separately.

What calibration does the harness report?

Because Jev returns a probability per option, it reports expected calibration error and a 4-way Brier score alongside accuracy; at 78.66% accuracy jev-latest reported 71.00% mean confidence, mildly under-confident.

CommunityJev Tools & CLIs

Jev Tmmluplus Eval

Harness that sits Jev for the 66-subject Traditional Chinese TMMLU+ suite, asking every item as one four-way A/B/C/D choice with calibration reporting.

See highlights

Open github.com

Stars: 3
Language: Python
License: MIT
Published by: Community
Author: lianghsun
Last verified: Oct 2026

lianghsun/jev-tmmluplus-eval

User repository on GitHub

View on GitHub

Evaluate TypeSafe AI's Jev (System One Model) on TMMLU+ v1.1 — four-way choice via the API's own response schema, 100% parse rate by construction

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Sit Jev — TypeSafe AI's System One Model — for TMMLU+ v1.1, the 66-subject Traditional Chinese benchmark, and score it the way the official leaderboard does. Every question is put to Jev as one four-way choice (A/B/C/D). Jev is not a chat model. You hand it a state plus a map of typed questions, and it returns one typed answer each — with calibrated probabilities and no generated text.

Quoted from the project's README. Read the full README.

Tagscommunity github

Works forJev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Maps every question to one four-way choice with A/B/C/D criteria, returning choice plus per-option probabilities.
CLI covers subjects, models, resumable runs, retries for failed items, and offline rescoring from JSONL.
Scoring mirrors the ievals leaderboard method with category means, micro accuracy, ECE, and Brier score.
Reports per-subject strengths and weaknesses, including weak arithmetic and strong language and medical items.

Watch out

Scores follow the pinned dataset revision and scoring rules in the repo, so compare only against the same revision and methodology. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

paper-radar-jev

Python CLI that pulls recent arXiv papers, scores relevance with TypeSafe and writes daily Markdown and JSON rankings with resume-safe storage.

Jev Tools & CLIs#community#github

Community

LYchoon/paper-radar-jevPython

GitHub-Issue-Classification-Using-Jev

GitHub Action triage that asks eight Jev questions in one call, gates labels in plain Python, and layers regex redaction, a Jev guard battery, and a write guard.

Jev Tools & CLIs#community#github

Community

KalyanM45/GitHub-Issue-Classification-Using-JevPython

jev-seo

547

Live SEO audit that crawls a site from one homepage URL, checks 52 rules tied to Google Search Central, and asks Jev typed questions about every page to rank fixes.

Jev Tools & CLIs#community#jevbest#catalog-import

Community

AgriciDaniel/jev-seoPython

SmartMoney-Cub

Local-first trading journal and review harness that is read-only over markets, captures runs into frozen envelopes and evidence packs, and requires explicit human promotion before any rule returns.

Jev Tools & CLIs#community#jevbest#catalog-import

Community

myc0576/SmartMoney-CubPython

jevclip

Video tool where Jev scores subtitle segments and code assembles a short highlights cut plus a de-watered full cut with timestamped summaries.

Jev Tools & CLIs#community#jevbest#catalog-import

Community

cclank/jevclipPython

jev-yaba-wechat

macOS floating WeChat companion that reads the visible chat with on-device Vision OCR, judges intent and reply risk with Jev, drafts toned replies with GPT, and only fills text after you approve it.

Jev Tools & CLIs#community#jevbest#catalog-import

Community

wuxie888/jev-yaba-wechatPython

From the Jev guides

Original write-ups on what this entry is for.

Jev on the Gateways: Vercel, Workers AI, and OpenRouter

Oct 12, 2026

The request shape barely changes across surfaces; the surrounding contract does. When to call the API directly and when a gateway, a Worker binding, or a router entry earns its extra hop.

Jev Tools & CLIs#gateway#vercel#cloudflare

Read article

Exporting Decision Models: GGUF, ONNX, and Device Runtimes

Oct 10, 2026

Taking a decision checkpoint out of Python: what the GGUF and browser WebGPU exports preserve, what they change, and the parity evidence the model cards publish.

Jev Tools & CLIs#open-models#local#onnx

Read article

Records at Rest: Documents, Inboxes, and Risk Queues

Sep 28, 2026

Jev labels a record that already exists. Code moves, flags, or holds it. How the document, mail, and transfer tools keep the action out of the model.

Jev Tools & CLIs#classification#email#security

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai