Article
CommunityJev Benchmarks & Evaluations
Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Collects 702,000 judgments across six datasets with 9,000 multiply annotated items.
- Jev's second-position probability advantage reaches 6.11 percentage points on summaries.
- Reversing reader endorsements changes Perplexity's selected humor text on 40.40% of pairs.
- Rejection framing reduces Clef's summary agreement with human preferences by 23.36 points.
- Rating formats shift score levels while often preserving rankings.
Questions about this entry
Short answers drawn from the linked project or page.
It audits presentation sensitivity in typed decision models, including Jev, across native choice and score interfaces. The study varies positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics while preserving the evaluated texts, then measures how selected choices, scores, and probabilities respond.
The audit covers six datasets containing 9,000 multiply annotated humor, helpfulness, and summarization items, and collects 702,000 judgments under the tested interventions.
Interventions varied positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics. The evaluated texts themselves were preserved, so measured shifts trace to presentation rather than content changes.
On summaries, Jev's second-position probability advantage reaches 6.11 percentage points, meaning the probability assigned shifts with the position in which an option is presented.
No. The findings conclude that repeatability, presentation robustness, and human alignment emerge as distinct properties of an evaluator, so strong performance on one does not imply the others.
The audit compares Jev with Clef Flash and Perplexity Decider. Reversing reader endorsements changed Perplexity's selected humor text on 40.40% of pairs, and rejection framing reduced Clef's summary agreement with human preferences by 23.36 points.
The paper states that reliable use requires validating the decisions and score interpretations on which an evaluation pipeline depends, since rating formats can shift score levels even when rankings are often preserved.
More Jev resources like this
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting a bias audit protocol for System One decision models, covering BBQ, CrowS-Pairs, and SilverSpoon adapted to the typed interface, twelve models plus a no-thinking ablation, and 1,638,000 decisions with swapped options.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.