Article
CommunityJev Benchmarks & Evaluations
Predicting What Answer Engines Cite: Open Decision Models, Rerankers and a Distilled 4B Student
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Ten engines answer GEO-Bench queries with the same candidate sources placed in their prompt.
- Where several engines answer the same query, they agree at a within-group Spearman correlation of 0.704.
- Jev and two open 27B decision models are the strongest untrained scorers at 0.575 to 0.578.
- GEO Citation Ranker 4B, a LoRA-tuned Qwen3-Reranker-4B, reaches 0.542 to 0.549 across three seeds.
- Its differences from the 27B models and from Jev are not distinguishable from zero on 229 queries.
Questions about this entry
Short answers drawn from the linked project or page.
It measures how well different scorers rank candidate pages by the share of an answer they will receive as citations, using GEO-Bench queries answered by ten engines that all receive the same candidate sources in their prompt.
It is a Qwen3-Reranker-4B fine-tuned with low-rank adapters on teacher labels for GEO-Bench queries other than the evaluation set. It reaches 0.542 to 0.549 across three seeds, and its differences from the 27B models and Jev are not distinguishable from zero on 229 queries.
Used without training, Jev is among the strongest scorers at 0.575 to 0.578, alongside two open 27B decision models. Most of the remaining distance between the distilled student and Jev disappears when Jev scores one passage at a time, as the student does.
GEO-Bench is the query set used for evaluation. Ten engines answer those queries under controlled retrieval, receiving the same candidate sources in their prompt, which lets the paper compare scorers against observed citation shares.
The authors release the distilled model, the benchmark, the predictions of every reported scorer, and the code.
No. Students trained on teacher labels and on observed engine citations reach the same level, and labels from Jev alone match the two-teacher target at 0.6B.
The provided material does not state a license for the released model, benchmark, predictions, or code, so terms would need to be confirmed from the paper or its repository.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper measuring gender bias in decision models, testing TypeSafe's closed Jev and five open models on gender-swapped yes/no decisions, BBQ, StereoSet, and Fulgu and Capraro's gender-bias studies, with results on top-answer flips and confidence-gate effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.