Article
CommunityJev Benchmarks & Evaluations
Fast, Cheap, and Fair? Evaluating Decision Models
arXiv paper presenting the first systematic bias and fairness audit of Jev, a typed-decision model for recruitment. It measures how demographic attributes and candidate cues shift screening scores, rankings, and shortlisting, against a noise floor, plus validity, latency, and cost.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- First systematic bias and fairness audit of Jev for recruitment decisions.
- Measures effects of explicit demographic attributes and implicit candidate cues.
- Compares effects against a noise floor from name substitutions and typographical variations.
- Reports output validity, latency, and cost alongside fairness results.
- Releases counterfactual CV datasets and evaluation code for reproducibility.
Questions about this entry
Short answers drawn from the linked project or page.
It is the first systematic bias and fairness audit of Jev, a typed-decision model, in recruitment applications, examining how inputs influence screening scores, rankings, and shortlisting outcomes.
Screening scores, rankings, and shortlisting outcomes, along with output validity, latency, and cost.
Effects from explicit demographic attributes and implicit candidate cues are compared against a noise floor derived from name substitutions and typographical variations.
The paper notes the EU AI Act classifies recruitment AI systems as high-risk, imposing strict requirements on their deployment, which motivates fairness evaluation.
Yes. The paper states that counterfactual CV datasets and evaluation code are released for reproducibility.
The paper frames compact System 1 decision models as a fast and cost-efficient alternative to general-purpose LLMs, while noting their fairness remains largely unexplored.
It targets recruitment applications of Jev and reports fairness, validity, latency, and cost for that setting; it does not cover other decision domains or deployment contexts.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Agent Handoff Gate: agents verify what they hand off
X: Jev as an LLM guardrail inside Agentgateway
X: Vercel's fx safety reviewer, measured at 18× faster
Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.
More Jev resources like this
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting Slot Attack, a black-box attack that appends text to evidence to steer decision models toward an attacker-chosen wrong option. Evaluates seven decision models on 16 Decision Bench tasks across seven domains.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv paper proposing OLIVIA, a protocol for look-ahead-free tests of AI decision models, applied to probability-returning models reading S&P 500 news with masked company identities, validation-only selection, and a hash-frozen analysis plan.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper evaluating 22 hosted decision models and two prompted general models as runtime classifiers for a multi-rule coding-agent policy engine. It reports recall, false-positive, latency, and cost results on a 2,643-call synthetic dataset with a sealed holdout.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.