Article
CommunityJev Benchmarks & Evaluations
Can Jev Improve Demonstration Retrieval for In-Context Learning?
arXiv paper testing whether Jev can rank candidate demonstrations for in-context learning by judging their utility for a given query, and distilling that judgment into a lightweight MLP to cut the cost of large-scale retrieval.
- Format
- Article
- Reading time
- 1 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Jev assesses the utility of candidate demonstrations for a given query.
- Each candidate requires an additional decision-making inference, adding overhead at scale.
- A knowledge distillation approach transfers Jev's utility estimation to a lightweight MLP.
- The distilled model ranks demonstrations efficiently while keeping competitive retrieval performance.
Questions about this entry
Short answers drawn from the linked project or page.
It asks whether Jev can improve demonstration retrieval for in-context learning by directly assessing the utility of candidate demonstrations for a given query.
Jev is described as a decision-oriented language model capable of producing probabilistic judgments, which the authors use as a new perspective on evaluating demonstration utility.
Applying Jev to large-scale retrieval introduces substantial computational overhead, because each candidate demonstration requires an additional decision-making inference.
It proposes a knowledge distillation approach that transfers Jev's utility estimation capability to a lightweight multilayer perceptron, which learns to approximate Jev's probabilistic judgments.
The paper reports that the distilled model enables efficient demonstration ranking while maintaining competitive retrieval performance.
It situates itself in prior work on retrieval strategies that identify informative demonstrations to improve downstream in-context learning performance.
It highlights the potential of decision-oriented language models for demonstration selection and offers a framework for integrating their decision-making capabilities into in-context learning.
More Jev resources like this
arXiv paper testing whether compact structured memory improves a probabilistic decision model's answers on 30 fully synthetic clinical cases across twelve domains, evaluated with 120 typed questions and reporting accuracy, log loss, Brier score, token and latency costs.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing GDGraph, a guideline-derived decision graph framework that compiles radiology guidelines into an executable graph and has a System One model answer localized clinical questions, reporting 90.8% accuracy across 347 cases.
Jev Benchmarks & EvaluationsArticle#arxiv#research#evaluation
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.