Question 1

Are these numbers real, or is Jev just judging itself?

Accepted Answer

The circularity is measured rather than waved away. The same comparison reads +0.053 NDCG@10 under Jev-only labels, +0.012 under merged labels, and -0.028 under LLM-only labels, the set Jev had no part in. Labels come from Jev plus claude-haiku-4.5 at temperature 0, agree within one level on 98.0% of pairs, and a 30-query hand-adjudicated subset agrees with the full set within 0.02 for every system.

Question 2

So should I use Jev as my reranker?

Accepted Answer

Not as a standalone one. jev-score(bge-m3@30) over bge-m3 is +0.012 NDCG@10 with a confidence interval of [-0.013, +0.037] and turns negative under LLM-only labels. What survives is fusion: rrf(bge-m3, jev@30) is +0.090 [+0.077, +0.104] and still +0.064 under LLM-only labels, for roughly $0.0002 per query.

Question 3

Why did reranking the shipped keyword list make results worse?

Accepted Answer

Recall, not ordering. ash's recall@10 for relevant items is 0.497 against bge-m3's 0.708, and ash returns nothing at all for 3 of the 164 queries, so reranking its top 30 with bge-m3 scores -0.026. On that weak lexical list Jev is the better reranker: jev-score(ash@30) beats bge-m3(ash@30) by +0.060, or +0.037 under LLM-only labels.

Question 4

What are these relevance labels actually grading?

Accepted Answer

Metadata-level relevance only, from the name, description and tags the systems themselves see, not whether a skill actually works. Both judges are language models, and there is no human-labelled subset beyond the 30 hand-adjudicated pairs. The pool is the union of four retrievers' top-30 capped at 60 per query, so a system retrieving outside the pool would be scored as zero.

Question 5

How were the relevance labels produced?

Accepted Answer

Two independent judges graded the pooled pairs: Jev, with one score question per candidate and 25 candidates per call, and Claude Haiku 4.5 at temperature 0 per pair. Agreement was 53.1% exact and 98.0% within one level (quadratic-weighted kappa 0.71), and a 30-query core subset was hand-adjudicated.

Question 6

What is the takeaway for adding Jev to a search stack?

Accepted Answer

Jev is worth adding only on top of semantic candidates and only fused: the reciprocal-rank fusion of bge-m3 and Jev gained +0.090 NDCG@10 over bge-m3 for about one $0.0002 call per query (328 rerank calls cost $0.066 here). As a standalone reranker it did not beat bge-m3.

Question 7

What did the judge-circularity check show?

Accepted Answer

Because Jev was both labeller and system under test, every system was re-scored under three label sets. The apparent Jev rerank advantage read +0.053 under Jev-only labels, +0.012 under merged labels and -0.028 under the unrelated-model-only labels, so only the fused result survives.

Jev Search Rerank Eval

About this project

Highlights

Quickstart

Questions about this entry

Reactions & coverage

From the Jev guides

From the Jev AI model community