Article
CommunityJev Benchmarks & Evaluations
Label-Wording Augmentation in a Typed-Decision Encoder
arXiv paper comparing two releases of Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks, to test whether an augmentation that presents labels in several surface forms helps when callers supply their own option wording.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Compares two releases of Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks.
- Evaluates ten label vocabularies over eighteen routing labels, with five candidates per question and, for eight vocabularies, eighteen.
- The augmented release is more accurate on ordinary or borrowed words in every setting, by up to 0.24; eleven of fourteen gains survive a Holm correction.
- On abbreviations and opaque codes there is no usable gain; on opaque codes both releases stay at or below chance.
- On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points; four evaluation hazards are reported.
Questions about this entry
Short answers drawn from the linked project or page.
It compares two releases of a typed-decision model to see whether training with labels presented in several surface forms improves accuracy when the caller, not the model developer, chooses how options are worded.
Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks. The later of the two releases adds the label-wording augmentation.
Ten label vocabularies were produced by three separate processes and checked for overlap with the training data. Questions were held fixed and only the wording of eighteen routing labels was varied.
On held-out vocabularies made of ordinary or borrowed words, including English-derived jargon, office slang and department names, the augmented release is more accurate in every setting, by up to 0.24.
On abbreviations it is not better and is worse on the clean slice. On opaque codes both releases stay at or below chance once labels related to training options are removed.
In most cells where accuracy rises the risk-coverage area falls, but with eighteen candidates confident errors rise on jargon and slang. Slang with eighteen candidates did not survive a Holm correction.
Gains do not transfer to abbreviations or opaque codes, and on two datasets neither release had seen, accuracy moved by -8.4 and +5.6 points. The paper also reports four evaluation hazards from comparing the two releases.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Independent audit of a frozen System One model on 142,658 human-labeled comments, benchmarked against 14 frontier reasoning models. It reports AUROC 0.83, ~USD 0.000025 per call, ~100 ms, and separates human disagreement from decision risk.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Hugging Face dataset of synthetic typed-decision records for Jev-style control and probability-estimation tasks. It holds 12 subsets with question, options, target, state_json, and metadata_json fields across train, calibration, validation, test, and ood splits.
Jev Benchmarks & Evaluations#open-models#calibration#evaluation
Community
Retrospective study comparing JEV, a non-generative decision model returning native option probabilities, with schema-constrained Qwen Flash on 2,048 patient-condition pairs from TrialGPT and MIMIC-III/IV under Zero Data Retention policies.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.