Question 1

Are the vendor's 193.6x speed and 444.6x cost claims real?

Accepted Answer

Both multipliers are real and both fall far short of the launch numbers. Measured on this task Jev was 3.25x-3.62x faster than Claude Sonnet 5 at p50 and about 40.6x cheaper per call, roughly 50x smaller on speed and 11x smaller on cost than claimed. The likely reconciliation, which the repo does not verify, is that the vendor figures compare a multi-step agentic pipeline against a single classification call.

Question 2

Is Jev more accurate than a frontier LLM on this task?

Accepted Answer

No — they tie exactly, 55/60 for Jev, jev-preview and Claude Sonnet 5, with the identical 100% clear, 71.4% ambiguous and 91.7% adversarial split on all three. The author reads that as the strongest evidence in the repo that task difficulty rather than the model set the ceiling. Notably, a narrow structured decision does not obviously need a frontier LLM at all, which is the practical finding.

Question 3

Is the confidence score safe to build an escalation path on?

Accepted Answer

On this task set, yes, and that is the result the repo exists for. Across 180 calls nothing ever said 1.000, or Claude's practical ceiling, and was wrong; Jev's misses ran 0.130 to 0.785 and Claude never went below 0.5 on anything. Two limits keep it honest: 50 of 60 predictions sit in the top confidence bin, so report the occupancy, and an easy sample saturates at 1.000, which is why calibration numbers over undemanding sets are worthless.

Question 4

Can I run this myself and disagree with the labels?

Accepted Answer

Yes, that is the design. The task set is published in tasks.jsonl with a difficulty field, the harness is about 200 lines, analyze.py computes accuracy by difficulty, latency percentiles and ECE with bin occupancy, and raw per-call output is committed under results. You need both a TypeSafe key and an Anthropic key, and there are no dependencies beyond the Python standard library. --repeat N separates latency variance from answer stability.

Question 5

What task does the benchmark actually test?

Accepted Answer

Classifying an agent tool call as readonly, destructive, privileged or exfiltration. The task set is 60 hand-labelled cases: 34 clear, 14 ambiguous and 12 adversarial, where the adversarial cases wrap a risky call in benign operational language such as routine cleanup before kubectl delete namespace prod.

Question 6

How close did Jev and Claude Sonnet 5 come?

Accepted Answer

They tied exactly on accuracy: 55 of 60 for jev-latest, jev-preview and claude-sonnet-5, with the same 100%, 71.4% and 91.7% split across the clear, ambiguous and adversarial slices. The difficulty of the task, not which model answered, set the outcome. Jev was 3.25x to 3.62x faster and about 40.6x cheaper on measured cost.

Question 7

What did the calibration table show?

Accepted Answer

Every incorrect answer on every backend came with hedged confidence. Jev's misses ran from 0.130 to 0.785, while Claude never expressed confidence below 0.5 on anything in the set. In the 0.9 to 1.0 bin Jev held 50 of 60 predictions at 98.0% accuracy, so most of the ECE figure is determined by that one bin.

Jev Benchmark — themsquared

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community