Which models does the retrieval audit cover?

It audits request independence in seven open Jev-style checkpoints, including Nimble, Von, and Laya-typed, evaluated on constructed three-passage states and TREC retrieval queries.

How were the audited test states constructed?

The authors built 276 three-passage states from 92 TREC retrieval queries, then applied paired interventions that separately change the requested questions, the displayed evidence, and candidate eligibility.

How often does request expansion flip Nimble's binary judgments?

Request expansion induces strict binary flips in 9.1% of Nimble's judgments in TREC DL2019 and 8.0% in TREC DL2020, at a threshold of 0.5 fixed before inference.

What happened to selection-or-abstention actions in the audit?

They changed in 21.4% and 18.9% of cases in TREC DL2019/2020 respectively, with correct-action rates changing by 0.0 and +8.2 percentage points and 95% query-bootstrap intervals that include zero for the first year.

Where do request-set flips come from, according to the audit?

Field masking traces all 70 request-set flips to the schema fields that are serialized into each scoring prompt, linking the instability to the prompt's field content rather than the question alone.

Did the audit reproduce on other checkpoints?

In Von, fixing a shared separator's position eliminates two eligibility-induced preference flips in selected diagnostic states; Laya-typed's default CUDA path produced five passage swaps without strict binary flips.

What does the audit imply for retrieval practitioners?

It questions the assumption that request independence holds in practice: the audited checkpoints' judgments change when the displayed evidence or requested question set changes, even though the documented interface evaluates typed questions independently.

CommunityJev Benchmarks & Evaluations

The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval (arXiv 2610.14631)

An audit of request independence in seven open Jev-style checkpoints, using 276 three-passage states built from 92 TREC retrieval queries. Paired interventions show request expansion flips 9.1%/8.0% of Nimble's judgments and changes up to 21.4% of selection-or-abstention actions.

See highlights

Open arxiv.org

Format: Article
Published by: Community
Last verified: Oct 2026

Tagscommunity research arxiv retrieval auditing calibration

Works forJev for Ranking Jev for Evaluation Jev for Verification

Used inJev for Software & DevTools Jev for Data & Analytics

Highlights

Audits request independence in seven open Jev-style checkpoints over 276 three-passage states from 92 TREC queries.
Request expansion induces strict binary flips in 9.1%/8.0% of Nimble's judgments on TREC DL2019/2020.
Selection-or-abstention actions change 21.4%/18.9% at a 0.5 threshold fixed before inference.
Field masking traces all 70 request-set flips to schema fields serialized into each scoring prompt.
Von's two eligibility-induced preference flips disappear when a shared separator's position is fixed.

Watch out

arXiv preprint dated October 10, 2026 (cs.IR, cs.CL), 15 pages, by Jianbo Chen. Findings come from constructed diagnostic states and one auditor's intervention design, so flip rates describe the tested checkpoints and prompts rather than Jev-style models in general.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

A pre-registered evaluation of Jev as a memory-retirement gate deciding when a new utterance retires an old stored claim. On 500 labeled test pairs, the binary gate retires 39.2% of claims whose change is only suggested; a three-way contract cuts that to 15.5% at a 15.3-point detection cost.