Article
CommunityJev Benchmarks & Evaluations
RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge
arXiv paper introducing RubriX, a rubric judge built from a LoRA-adapted encoder and a softmax over scaled cosine similarities rather than text generation. Reports 75.4% exact accuracy on held-out rubrics and roughly 28 ms cached judgments.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- A LoRA-adapted encoder embeds the input and each level description; a softmax over scaled cosine similarities gives a probability per level.
- RubriX reaches 75.4% exact accuracy on held-out rubrics, against 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge.
- Within 6 percentage points of Jev on most of eight human-labelled benchmark sets, with Jev holding a larger lead on two.
- With level embeddings cached, latency stays near 28 ms per judgment as level descriptions grow from 10 to 3,000 tokens.
- At 3,000 tokens per level RubriX is 7.7x faster than Jev and 18.8x faster than the Qwen3-8B judge.
Questions about this entry
Short answers drawn from the linked project or page.
RubriX reaches 75.4% exact accuracy on held-out rubrics. The paper reports 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge on the same comparison.
On eight human-labelled benchmark sets RubriX is within 6 percentage points of Jev's accuracy on most of them. Jev keeps a larger lead on two sets and ranks responses best on average.
With level embeddings cached, latency stays at about 28 ms per judgment as level descriptions grow from 10 to 3,000 tokens. Jev rises from 44 ms to 211 ms and the Qwen3-8B judge from 32 ms to 516 ms in that range.
No. A LoRA-adapted encoder embeds the input and each level description, and a softmax over scaled cosine similarities gives a probability for every level, so no generative call is needed per judgment.
RubriX was trained on public rubric data and proprietary synthetic data. Because part of the training set is proprietary synthetic data, the full training mixture is not public.
Against the results reported in the G-Eval paper, RubriX outperforms G-Eval with GPT-3.5 on every SummEval dimension and on both QAGS subsets.
Jev keeps a larger accuracy lead on two of the eight human-labelled benchmark sets and ranks responses best on average, even though RubriX is faster once rubric level embeddings are cached and the level descriptions grow long.
More Jev resources like this
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.