Jevals — Independent Jev Benchmark for Models That Decide

Mixed, and the site is upfront about it. On the noul board Gemini 3.8 Flash leads at 73.0 against Jev's 69.0, and on Banking77 choice it leads 74.1 to 67.8 while DeepSeek V4.1 Flash trails at 63.9. Jev's actual headline is narrower: on nouls it is reported statistically tied with the best of six LLMs at roughly 1/28 of the price.

Independent on its face and explicit about the relationship: the site states it is not affiliated with TypeSafe AI or Jev and does not represent them, publishes data under CC-BY-4.0 with an API and feed, and says every row is run by them at list price. The catch is coverage, since the most-cited frontier models including GPT-6 Astra and Claude Fable 5.1 are listed as coming next rather than scored.

Very little, and the board says so. The scale sets 100 as perfect and 0 as guessing the label base rates, so 9.2 is barely above chance. Jev leads the listed models at 9.2 against GLM-5.3 at 7.8 and Mercury 2.5 at negative 5.5, but the page states plainly that no model clearly beats guessing here, which makes it the weakest board for ranking.

Partly. Data, methodology, a changelog and an API are published, with the underlying data on GitHub under CC-BY-4.0, and each board is 300 questions scored against human labels rather than another model's opinion. The landing page does not restate enough of the harness to re-run a row yourself, so plan to bring your own runner if you need per-item results.

The three boards cover 300 questions each: PubMedQA nouls, Banking77 choices and HelpSteer2 helpfulness scoring, which is 31,500 scored decisions across three question types and seven models, all measured against human labels.

The published data is CC-BY-4.0 and the site exposes an API and a feed, with the data also on GitHub, so you can reuse and cite it; the source datasets keep their own licences.

The scored rows are hosted Jev plus six LLMs including Gemini 3.8 Flash, GLM-5.3, DeepSeek V4.1 Flash, Qwen3.8 Flash, Mistral Medium 3.5 and Mercury 2.5; frontier models like GPT-6 Astra are listed as coming next rather than scored.