CommunityJev Benchmarks & Evaluations
Laya vs Jev — huggingface.co
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
- Format
- Docs
- Published by
- Community
- Author
- Luni
- Last verified
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is Laya vs Jev — virajbhartiya.
- Languages
- Python
- Capabilities
- Open source
Highlights
- Raw Laya sits at chance on phishing (0.505 accuracy, 1.2% recall) with AUROC 0.678 against Jev's 0.689; Platt scaling lifts accuracy to 0.611.
- Laya fine-tuned on typed-decisions reaches 0.767 against Jev 1.13.0's published 0.727 and the 0.735 teacher self-agreement ceiling.
- Latency on an RTX 5090 in fp16: 10.7 ms for one question, 496 ms for 100 in one pass and 14.9 s cold load.
- The eleven-assertion probe suite catches 7 failures for base Laya, including P(needs a human) plus P(a bot can resolve it) at 0.09.
- Code and results are Apache 2.0; the Luni/laya-grounded checkpoint is CC-BY-NC-4.0 because its mixture includes facebook/anli and customer-support-tickets data.
Watch out
Code and results are Apache 2.0, and the card warns its probe suite is not a benchmark: eleven self-written assertions over two examples. ECE is computed against argmax correctness so soft targets inflate it, option-renaming stability worsens after fine-tuning, the fine-tune also makes phishing worse, and the checkpoint's CC-BY-NC-4.0 terms cap commercial reuse.
Questions about this entry
Short answers drawn from the linked project or page.
A measured Laya-versus-Jev comparison run on one RTX 5090: phishing on AreLit/PhishNChips, the LocalLLaMA/typed-decisions suite, latency scaling, and an eleven-assertion probe suite, with raw logs under results/.
Raw Laya sits at chance: 0.505 accuracy, ECE 0.441, AUROC 0.678 and 1.2% recall, saying 'not phishing' to almost everything; fitting a Platt bias term on 1,000 emails lifts accuracy to 0.611.
0.767 accuracy against Jev 1.13.0's published 0.727, but the card notes it also beats the 0.735 teacher self-agreement ceiling, the limit of real label signal, so above that line it is memorising noise.
In fp16 after warmup: 10.7 ms for 1 question, 42.6 ms for 10, 246 ms for 50 and 496 ms for 100 in one pass, with a 14.9 s cold load.
Laya fails 7 of 11 assertions on a support ticket and a phishing email, including P(needs a human) plus P(a bot can resolve it) at 0.09 when it should be near 1.0.
On 180k public items for three epochs the consistency-loss arm reaches 0.676 on typed-decisions and 0.840 held-out accuracy, but phishing gets worse (0.512 against 0.611) and option-renaming stability drops.
Code and results are Apache 2.0, with parts of bench/ derived from the Apache-2.0 NandhaKishorM/laya project; the Luni/laya-grounded checkpoint is CC-BY-NC-4.0 because its mixture includes facebook/anli and customer-support-tickets data.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
3GitHub stars
The first Chinese-language calibration benchmark for Jev-class System One decision models: 378 items and 467 questions across voice routing, e-commerce service and content moderation, with accuracy, ECE, Brier, NLL, AUROC, option-order and zh-CN/zh-TW robustness metrics.
Jev Benchmarks & Evaluations#community#benchmark#calibration
Community
CodyQin/zh-decision-benchPython
CodyQin/zh-decision-benchPythonReal human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.