Jev directory
CommunityJev Benchmarks & Evaluations
JevBench: jev vs gev-1.0 — AnyEval
AnyEval's paired JevBench comparison of jev against gev-1.0 on 231 shared problems: the win table, an exact McNemar result, and end-to-end answer times for both models measured in one fixed stratum.
- Reading time
- 3 min
- Published by
- Community
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
Highlights
- 231 shared problems, one draw per problem per model, all in one stratum: scaffold baseline@v1, 16000-token limit, no judge on record.
- Win table: jev right on 199 problems and gev-1.0 right on 200; the 19 discordant problems split 9 versus 10.
- Exact McNemar gives p = 1.0000, and the page labels the test unadjusted and exploratory.
- Speed: jev median 2.04 s and p90 5.93 s against gev-1.0's 1.99 s and 6.59 s across the 231 timed problems.
- Each separating problem links both traces and a step-by-step diff of the two runs.
Watch out
Grounded on the captured page text of this exact versus view on 2026-10-15; the digest does not include the JevBench eval's own description page, and the page itself notes the test covers only the pair chosen, with an unadjusted exploratory result. The timings are observed for these draws, not a guarantee for future requests.
Questions about this entry
Short answers drawn from the linked project or page.
Two models on the same JevBench problems: jev, served by typesafe, against gev-1.0, served by trustedrouter. It is one draw per problem per model, and the problems where exactly one model was right carry all the evidence in an exact paired test.
Across 231 shared problems, jev was right on 199 and gev-1.0 on 200; 190 problems were right for both and 22 wrong for both. The 19 discordant problems split 9 to jev and 10 to gev-1.0, and the exact McNemar test gives p = 1.0000.
Every pair was measured under scaffold baseline@v1, a 16,000-token limit, and judge none on record, with jev served by typesafe and gev-1.0 by trustedrouter. Runs under other pins are treated as different measurements and are not pooled into these counts.
Over the 231 timed problems: jev's median was 2.04 s with a p90 of 5.93 s, and gev-1.0's was 1.99 s with a p90 of 6.59 s. Jev was faster on 154 problems and gev-1.0 on 77, a median paired difference of -0.524 s.
No. They run end to end from AnyEval's worker through the gateway, so gateway overhead is included and the host's raw latency is not what is shown; safety monitors and harness overhead are excluded. The page calls them observed timings, not a guarantee for future requests.
Yes. The page lists each of the 19 discordant problems with the verdicts and links to both runs plus a step-by-step diff, so the evidence behind the count is visible rather than just the totals. Examples include hard-opus-a-probability-07 for jev and hard-opus-c-long_policy-03 for gev-1.0.
No. The page itself flags the test as unadjusted and exploratory: it covers the one pair you chose, and choosing that pair after scanning a table is multiple testing that no correction here can undo. Comparing two rates would be hopeless at this sample size; the paired test works only because discordant problems carry the evidence.
More Jev resources like this
Benchmark Heaven's live JevBench board for typed decision models: florianstandhar's reproducible benchmark, presented with frozen versioned rankings, price bases, hosting and confidentiality filters, and per-model evidence requirements.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
A Convex page for comparing models that answer verified multiple-choice questions about Convex, scoring them with source-weighted results and showing latency, cost and inspectable outcomes alongside the scores.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonIndependent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonA minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonBenchmark comparing TypeSafe Jev 1.13 with the local Laya MLX typed-decision model across 1,470 synthetic items and eight tasks: accuracy, calibration, latency and throughput, with scripts, datasets, predictions and the full report in the repo.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
harrymunro/jev-laya-benchmarkPython
harrymunro/jev-laya-benchmarkPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.