User repository on GitHub
TypeSafe's Jev vs gpt-5.4-mini and gpt-5.6-luna on four public classification sets: cases, per-item answers, scoring, charts
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
Jev Eval — onlyoneaman
A measurement study that scores Jev against gpt-5.4-mini and gpt-5.6-luna on four public classification sets, then against recorded outcomes on a few thousand real pipeline decisions. Every per-item answer and the scoring code are committed.
- Stars
- 0
- Language
- TypeScript
- License
- MIT
- Published by
- Community
- Author
- onlyoneaman
- Last verified
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is Jev Eval — Shogo-nfrealmusic.
About this project
TypeSafe opened early access to Jev on 15 September. The headline numbers in the launch compare it with a frontier model writing text. Most teams do not run classification on a frontier model. The decisions a pipeline makes thousands of times a day (is this page worth reading, is this email a request, which bucket does this go in) mostly run on the small, fast tier already: flash-lite class models, gpt-5.4-mini, gpt-5.6-luna. That is the tier Jev has to beat, and the one I measured it against.
Quoted from the project's README. Read the full README.
Works forJev for Classification
Used inJev for Software & DevTools
- Languages
- TypeScript
Highlights
- Jev leads on three of four public sets: 98.7% on Enron spam, 95.7% on SST-2 sentiment, 91.3% on AG News topic.
- Banking77 intent is the loss: 76.0% against 81.7% for gpt-5.6-luna, with all 77 options in one question.
- Median call time is 0.8 to 0.9 s versus 1.4 to 4.2 s for gpt-5.4-mini, at 5 to 56 times lower cost.
- Deployed as a reject filter, replays of five production runs gave identical output with 50 to 67% fewer routing calls.
Watch out
The public-set comparison required every model, including the LLMs, to return a probability for every option, which inflates the baselines' output token cost and is why gpt-5.4-mini runs to several dollars per thousand calls on the 77-option set. TypeSafe does not document how confidence is computed, and the post notes it is not the top probability, so gating thresholds rest on an opaque number. Latency was measured with 20 calls in flight, and the author reports the tail is not always clean at 100 concurrent calls, so the sub-second figure is the pipeline case rather than the worst one. The accurate-when-confident column is a bin of the model's own confidence and is accompanied by the share of items landing in that bin, which is as low as 66% on Banking77. One routing row is scored against the existing classifier rather than an outcome, since those pages have none of their own. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
The repo is meant to be re-run. Every case, every per-item answer from all three models and the scoring code are committed, with a Reproduce section describing the procedure. The scale is about 16,000 calls across two days: 300 random items from each of Enron spam, SST-2, AG News and Banking77, plus a few thousand real pipeline decisions. You can check the numbers, at the cost of spending your own API budget reproducing them.
It is a bin of the model's own reported confidence: 0.9 or higher for a pick-one question, and P(yes) at or below 0.1 or at or above 0.9 for a yes or no. Each table reports both the accuracy inside that band and the share of items that fall in it, which is the number that keeps it honest - 90% of items on Enron spam, 82% on SST-2, 88% on AG News, 66% on Banking77. TypeSafe does not document how confidence is derived.
Two documented failures. On whole-document reads it is worse than the small model the author runs in production, and no prompt fixed that. On deciding whether a page carries a value for one of 135 report rows it agreed with the outcome 68.6% of the time, needed roughly twice as many extraction jobs to match recall, and its probability swung by 0.5 between runs on the same page. A general LLM failed the same way, so the post blames the task rather than the model.
It is measured under concurrency, which is the honest case, but it is not the whole picture. Median is 0.8 to 0.9 s and the 95th percentile 0.9 to 1.1 s with 20 calls in flight, and the slowest of 300 landed between 1.4 and 1.8 s. The author adds that the tail is not always clean: at 100 calls in flight on his own pages, a few per thousand took 10 to 35 seconds. There is no stated p99.9 figure.
About 16,000 calls over two days: 300 random items from each of Enron spam, SST-2, AG News and Banking77, plus a few thousand real pipeline decisions scored against what happened afterwards. Every per-item answer and the scoring code are committed.
TypeSafe does not document how confidence is computed and it is not the top probability (0.93 against 0.97 in one example); gating on the top probability instead gave the same picture within a point. Confident here means within 0.1 of an end for yes/no, or 0.9 or higher for a pick-one.
Latency was measured with 20 calls in flight, where Jev's median was 0.8 to 0.9 s with a 95th percentile of 0.9 to 1.1 s. At 100 calls in flight on the author's own pages a few per thousand took 10 to 35 seconds, so the sub-second figure is the pipeline case rather than the worst one.
More Jev resources like this
32GitHub stars
CLI that calibrates Jev questions against your own labels: write questions, label examples in JSONL, run check, and get a verdict per question — gate, gate-above-confidence, ranker, unusable, or too-few-examples.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
smkrv/jev-calibrateTypeScript
smkrv/jev-calibrateTypeScript7GitHub stars
Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
y0usaf/jev-lmTypeScript
y0usaf/jev-lmTypeScript30GitHub stars
Chat client over jev-latest, which cannot generate text: the transcript becomes the state and each step is a choice over a 20,000-word dictionary, so replies are built one word at a time.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
Bewinxed/jevgptTypeScript
Bewinxed/jevgptTypeScript3GitHub stars
Plugin for Claude Code and native Codex that layers Jev-assisted task routing, evidence search, context compaction and readable tool output on top of the host, where typed judgments inform policy but the host keeps ownership of permissions and execution.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
MongLong0214/jev-gateTypeScript
MongLong0214/jev-gateTypeScript0GitHub stars
A small Node.js TypeScript CLI for trying Jev on synthetic support text: one urgency question, three questions in a single request, or a latency comparison. No web server and no UI.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
Menny1337/jev-labTypeScript
Menny1337/jev-labTypeScript1GitHub stars
CLI that normalizes messy Japanese addresses to postal codes using Japan Post KEN_ALL and JIGYOSYO master data, asking Jev only for the steps rules and lookups cannot settle, and printing the probability behind each decision.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
smasato/jev-jp-addressTypeScript
smasato/jev-jp-addressTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.