Jev Rerank Bench

It established neither a winner nor equivalence. The rubric difference is +0.001 nDCG@10 with a 95% interval of -0.009 to +0.012, and weighting every query equally instead puts Cohere Pro ahead at 0.756 against Jev's 0.738. What Jev does lead on is negation, 71% of NevIR pairs right against Cohere Pro's 67% and zerank-2's 61%.

Treat them as exploratory. The intervals are paired bootstraps with no adjustment for the multiple comparisons in a table this wide, and the headline mixes eight English datasets with separately reported BRIGHT subsets, NevIR pairs and MIRACL French. Costs are eight-dataset rates averaged over 2,327 queries including queries with no relevant candidates, and the README states they are not invoice-verified or equivalent deployment costs.

Close, depending on the setup. Qwen3-Reranker-4B reaches 0.660 nDCG@10 against 0.670 for Jev asked one passage at a time, which is inside noise, yet it trails Jev's rubric setup and beats every Jev setup on French. Open-Jev 9B reads negation better than Jev asked the same way, 77% of pairs against 71%. Cost columns for self-hosted rows are GPU time, not an API price.

Batching wins. The 4-level rubric with all 30 passages in one call scores 0.692 at 422 ms, and 30 yes/no fields in one call scores 0.685 at 396 ms. Patterns that lost ground: one Choice plus none at 0.684, a batch-prune cascade at 0.674, yes/no per pair at 0.670 but 8.2 s, and 45 duels over the top 10 at 0.580. BM25 as the floor scores 0.486.

Models start with the same thirty BM25 candidates, each cut to 2,000 characters. The duel variant only compares the first ten, and the NevIR test supplies its own two-passage pairs rather than using the BM25 list.

Calls used jev-latest and reported version 1.13.0. Measurements began September 16, 2026, with the Qwen controls added September 17, and the report grades eight English datasets plus five BRIGHT subsets, NevIR and MIRACL French separately.

For self-hosted rows it is GPU time, not an API price: the seconds each call kept a rented RunPod card busy, times that card's hourly price, divided by the number of requests the card was working on at once. Setup and idle time are excluded.