CommunityJev Benchmarks & Evaluations
Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate (arXiv 2610.15710)
A pre-registered evaluation of Jev as a memory-retirement gate deciding when a new utterance retires an old stored claim. On 500 labeled test pairs, the binary gate retires 39.2% of claims whose change is only suggested; a three-way contract cuts that to 15.5% at a 15.3-point detection cost.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Test set: 500 pairs built from STALE, labeled by three LLM annotators and two human adjudicators.
- Of 179 pairs STALE marks changed, 72 meet the contract's bar for an established change.
- The binary gate retired no claims across 141 same-conversation non-changes.
- Binary gate prematurely retires 39.2% of suggested changes; three-way cuts it to 15.5% at a 15.3-point cost.
- A development-selected escalation cascade was indistinguishable from simply raising the threshold.
Watch out
arXiv preprint posted October 11, 2026 (cs.AI) by Xupeng Chen and Chenqian Le; 11 pages, 3 figures, 4 tables. Gold labels come from LLM annotators with two human adjudicators, one an author, and the held-out evidence is a single vault scored against a second annotator, so the numbers are not an independent replication.
Questions about this entry
Short answers drawn from the linked project or page.
Long-term assistants store claims about users, and a gate decides when a new utterance should retire an old stored claim. Existing benchmark labels only mark a pair as changed or not, so they cannot separate a gate acting on an established change from one acting on a hint.
It separates SUPERSEDED from NEEDS-CONFIRMATION and COMPATIBLE. Of the 179 test pairs STALE marks as changed, only 72 meet the contract's bar for an established change, which is why the authors argue the common binary label hides the distinction.
Jev never retired a claim across 141 same-conversation non-changes, but it retired 39.2% of claims whose change was only suggested. The paper names this failure premature retirement and treats it as the central risk of wiring a binary score to deletion.
Partly. It cut premature retirement from 39.2% to 15.5% but imposed a 15.3-point detection cost, missing the pre-registered 10-point budget, and the authors note it lands on the binary gate's own threshold frontier under three readings of the gold labels.
They built 500 test pairs from STALE, change pairs plus same-user controls, and labeled them with three LLM annotators and two human adjudicators, one an author and a second annotator, with provenance reported. Verdicts held under three readings of the gold labels.
On a held-out vault it performed no better than simply raising the threshold, with premature retirement of 26.8% versus 6.5% on development data. The paper attributes part of the gap to gold labels sharing errors with same-family escalation models and the rest to small-sample selection on development data.
Do not wire a binary score to delete: route suggested changes to pending evidence instead. The authors also warn against trusting a cascade selected on development data of this size, since the development-selected cascade was indistinguishable from a simple threshold change.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: jevmem keeps Claude Code project memory by asking Jev which turns are worth saving.
avinashjetwani61 pointsJevmem – automatic project memory for Claude Code, built on JevRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An audit of request independence in seven open Jev-style checkpoints, using 276 three-passage states built from 92 TREC retrieval queries. Paired interventions show request expansion flips 9.1%/8.0% of Nimble's judgments and changes up to 21.4% of selection-or-abstention actions.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScriptFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.