Jev directory
CommunityJev Benchmarks & Evaluations
JevBench — Benchmark Heaven
Benchmark Heaven's live JevBench board for typed decision models: florianstandhar's reproducible benchmark, presented with frozen versioned rankings, price bases, hosting and confidentiality filters, and per-model evidence requirements.
- Reading time
- 3 min
- Published by
- Community
- Author
- florianstandhar
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
Highlights
- Described as a reproducible benchmark for typed decision models, published as a ranked board of Jev-class models.
- Frozen, versioned ranking pages exist, such as v1.4.1, and the site labels itself BETA and work in progress.
- Price-basis options run from output only and input only through 1:1, 3:1, 10:1, 20:1, 30:1 and 100:1 blends, plus adjusted costs.
- EU hosting means inference runs inside the EU, checked per model against documentation; global deployments and EU billing alone do not count.
- Evidence requirements sit above the table: benchmark evidence, a priced provider, and measured task tokens.
Watch out
Grounded on the site's meta description and the captured page chrome on 2026-10-15; the capture is mostly interface text (menus, filters, definitions) rather than the leaderboard rows themselves, so this entry quotes no scores beyond the metadata's description and the compare view's summary. The digest's compare view against Hopper is a query variant of this same board and was not given a separate entry.
Questions about this entry
Short answers drawn from the linked project or page.
A live, ranked leaderboard for typed decision models, described in the site's metadata as florianstandhar's reproducible benchmark. It presents Jev-class models with frozen, versioned ranking pages, such as v1.4.1, and the site is marked BETA and work in progress.
The visible options include composite scores, a benchmaxxing signal, price bases, workload settings and several named ranking schemes. Evidence requirements such as benchmark evidence, a priced provider and measured task tokens sit above the table they apply to.
Price-basis options range from output only and input only through fixed input/output blends at 1:1, 3:1, 10:1, 20:1, 30:1 and 100:1, plus adjusted costs, and a task workload that can be held the same for every model or set as used on OpenRouter.
Inference must run in an EU region: an EU region, AWS Bedrock's EU cross-region profiles, Azure's Europe Data Zone, or a provider whose whole public fleet is documented as EU-hosted, checked per model. Global deployments, EU billing regions, EU companies and EU control planes do not count on their own.
The board includes a data-confidentiality column covering what a provider may do with your prompts, including whether it trains or keeps your data, and it offers open-weights-only and hide-deprecated filters alongside a one-variant-for-reasoning-models setting.
Yes, in a compare view of the same board: Benchmark Heaven scores Jev 1.13.0 against Hopper across four columns, and that view's metadata says Hopper wins three columns while Jev keeps Intelligence. That compare view was not given a separate entry because it is a query variant of this page.
The metadata credits florianstandhar's JevBench, and the site labels itself BETA, work in progress, with evidence requirements baked into the board. The captured page was mostly interface text, so this entry quotes no scores beyond the metadata's description and the compare view's summary.
More Jev resources like this
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonProphetLab's Texas Hold'em benchmark and playground for decision models: separate cash and sit-and-go leaderboards, hand replays, a Jev-backed advisor, and bring-your-own-agent tables, all with virtual chips.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
prophetlab/jevpokerbenchPython
prophetlab/jevpokerbenchPythonA minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonA benchmark for how efficiently decision models plan across multi-step problems: models that explore fewer unnecessary intermediate states score higher, and a perfect score means beelining along a shortest path. Seven domains, seeded trials and classic baselines.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
kyle-pena-nlp/beelinebenchPython
kyle-pena-nlp/beelinebenchPythonAnyEval's paired JevBench comparison of jev against gev-1.0 on 231 shared problems: the win table, an exact McNemar result, and end-to-end answer times for both models measured in one fixed stratum.
Jev Benchmarks & Evaluations#community#catalog-import#yibie
Community
A Convex page for comparing models that answer verified multiple-choice questions about Convex, scoring them with source-weighted results and showing latency, cost and inspectable outcomes alongside the scores.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.