CommunityJev Benchmarks & Evaluations
Jev Safety Bench
Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.
- Format
- Docs
- Published by
- Community
- Author
- SabaPivot
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 83,183 source requests in three choice variants, 249,549 rows total, with A/B meanings swapped in the reversed and serialization_control versions.
- Per-example results: 1,161,150 scored answers plus 352 unsupported records, joinable to requests by pair_id and variant.
- Original Jev identified 603/655 harmful OR-Bench Toxic requests (92.1%) against Laya's 273/655 (41.7%).
- On benign OR-Bench Hard, Laya scored 1,204/1,319 (91.3%) against Original Jev's 447/1,319 (33.9%).
- Laya's OR-Bench Toxic accuracy stayed 41.7% in both choice orders while 56 of 655 answers changed: 28 became correct and 28 incorrect.
Watch out
Licence 'other'. The card stresses these are classification results, not tests of refusal or deployment safety: 'type-safe' describes the output format, reversal changes both order and letters so it does not isolate order, only the original Jev was run on the order-only control, and partial repeats must not inflate primary scores.
Questions about this entry
Short answers drawn from the linked project or page.
XSTest, OR-Bench and HarmBench. Each source request ships in three choice variants, original, reversed and serialization_control, for 249,549 rows derived from 83,183 unique requests.
The answer texts swap between A and B; a separate order-only control keeps each letter's meaning but sends the choices in the opposite order, and only the original Jev was evaluated on that control.
No. The card says they are classification results, not tests of actual refusal or deployment safety, and that 'type-safe' describes the output format rather than decision correctness or safety.
Original Jev identified 603 of 655 harmful OR-Bench Toxic requests (92.1%) against Laya's 273 (41.7%), but on benign OR-Bench Hard Laya scored 1,204 of 1,319 (91.3%) against Original Jev's 447 (33.9%).
Laya's OR-Bench Toxic accuracy stayed at 41.7% in both orders yet 56 of 655 answers changed, 28 becoming correct and 28 incorrect; the card says the same score can hide different decisions.
Follow BENCHMARK_GUIDE.md for the prediction file format and score answers with tools/score_predictions.py; CONFIGURATIONS.md gives the exact questions, model formats and settings used in the release.
Repeats are partial at 246,313 rows, and a result is identified by (model, pair_id, variant, repeat_index) with repeat zero as the main evaluation, so the card says repeats must not inflate primary scores.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Building a contextual brand-safety scorer on Jev from scratch.
X: A website slop detector that checks 35 tells in 243 ms
X: Breaking, golden nugget, or slop
X: A filter for reply-guy comments
More Jev resources like this
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
6GitHub stars
Chinese-language research project for multimodal social-risk moderation: a local Qwen3-VL-32B with language-side LoRA emits structured evidence JSON, and a fixed Jev policy turns that evidence into risk bands, with frozen test leaderboards and pure-Jev ablations.
Jev Benchmarks & Evaluations#community#moderation#multimodal
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
25,418 Korean text-only evaluation items from 9 public datasets and 22 subsets, normalized to finite-choice decisions, with baseline results for ten model configurations.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev community
Posts from builders shipping with Jev right now.