Question 1

The report says the vendor's cost multiple is a peak number. What do other tests show?

Accepted Answer

Four independent tests measured speedups between 5x and 25x, and the most conservative of them found a cost advantage of 1.6x rather than several hundred times. The report's framing is that a multiple is only meaningful once you know what it is measured against and over which segment, which is also the reason it records a source tier for every number. It separately shows that the vendor's zero-hallucination framing means zero type errors — the model never returns an option outside the list — not zero wrong answers.

Question 2

How much weight can I put on the 95.6% Chinese accuracy figure?

Accepted Answer

It is 86 correct out of 90 judgments across 50 self-composed Chinese samples, with an ECE of 0.070, and the README states the gold labels were written by one annotator and locked before the run, with the file hash recorded. That supports one narrow claim: how the model behaved on those 50 samples. The gate result is thinner still — of 70 Noul and Choice judgments cut at 0.80 winning probability, 66 auto-passed with every one correct and the 4 routed to a human contained all 3 errors, which is promising but built on a handful of mistakes.

Question 3

Does the headline aggregate score measure human-level performance?

Accepted Answer

No, and this is one of the report's specific findings. The denominator behind that aggregate is the average of two frontier models, not human raters, so the number moves whenever the graders change. The report also traces a more widely circulated percentage to one number cherry-picked from each of four different workflows. It keeps that distinction visible by tagging every figure with a source tier — vendor documentation, vendor evaluation, independent test, community, own measurement, public news — and marking a figure verified only when the author read it in the raw material locally.

Question 4

Can I rerun the Chinese evaluation instead of trusting it?

Accepted Answer

The samples, scripts and raw returns for every call are committed. You set TYPESAFE_API_KEY, then run run.py for 200 calls at about four minutes and roughly $0.005, and analyze.py to write summary.md and metrics.json. Because the gold answers in samples.json were locked before the test started, swapping in your own samples and comparing becomes a fair test of your cases rather than a re-read of theirs.

Question 5

Is it new architecture or a fine-tuned encoder with a new name?

Accepted Answer

Unresolved, and the report is careful to say so rather than guess. It lays out the two hypotheses and names the three pieces of evidence that would separate them, noting that none of the three exist yet. The author states plainly that available public evidence cannot distinguish the two.

Question 6

What is in the report itself and under what license?

Accepted Answer

A 52-page PDF with 10 charts and 143 traceable numbers, free to download, with chapter sources, the data table, and the 50-sample Chinese evaluation committed alongside. The code is MIT, while the report text and data are CC BY-NC-SA 4.0. The author is unaffiliated with TypeSafe and self-funded the evaluation run at a total of $0.0046.

Question 7

What does the report say about latency from inside China?

Accepted Answer

That its latency figures are not a direct connection from mainland China. The run's exit IP was in Manila, so the recorded latencies describe that route rather than the one a reader in China would see. The cost and accuracy numbers are unaffected by geography; only the timing claim is.

Jev Report

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community