Post
CommunityJev Practices & Patterns
Reranking 33,047 catalog entries
A measured search reranking run over 33,047 catalog entries, 164 real queries, and 9,831 labelled pairs, reporting how Jev compares with BM25 and bge-m3.
- Format
- Post
- Reading time
- 2 min
- Published by
- Community
- Author
- GoSailGlobal
- Last verified
- Sep 2026
Highlights
- Covers 33,047 catalog entries, 164 real queries, and 9,831 labelled pairs.
- Compares Jev reranking against BM25 and bge-m3.
- Publishes the measurement setup rather than a single headline number.
Watch out
Author-run measurement on one catalog; the labelled set and prompts are not independently audited.
Questions about this entry
Short answers drawn from the linked project or page.
Barely, and not significantly. Reranking the top 30 bge-m3 candidates moved NDCG@10 by only +0.012 with a confidence interval that crosses zero. MRR and top-3 hit rate rose clearly, but the pattern is the strongest result being pushed to first place while the rest get reshuffled.
Yes, and he measured it. With Jev as the only judge its lead was +0.053; with two judges merged it fell to +0.012; with Haiku alone as judge Jev was behind by 0.028. Same comparisons, opposite conclusions, so he only stands behind the Haiku column for anything involving Jev.
It is a real gain for a trivial call. RRF-fusing the Jev and bge-m3 rankings reached NDCG@10 of 0.864, +0.064 to +0.116 over pure vector retrieval, holding under all three judges. The cost is one extra API call per query, about $0.0002, inside a run that cost $2.6 in total.
Yes, the data, graded labels and per-query scores are all open, so scoring recomputes without an API key. Two caveats before you trust the setup: the labels were partly produced with Jev itself in the judging loop, and his own hub's keyword ranking scored only 0.609 NDCG@10 because half the relevant results never entered the candidate set at all.
Jason Zhu, posting as @GoSailGlobal, ran it on his Agent Skills Hub catalog: 33,047 entries, 164 real queries in Chinese and English, and 9,831 graded pairs, for a total spend of $2.6. So it is one operator's self-test on his own index rather than a vendor or third-party benchmark.
A recall ceiling. The hub's live CLI keyword ranking scored only 0.609 NDCG@10 because about half of the relevant results never entered the candidate set at all, so no reranker could recover them. That is the author's own takeaway: reranking fixes ordering, not recall.
Worse for Jev. A reply from sean lee says their own reranking test found Jev still lagged SLM-based rerankers by a clear margin. It is an unverified one-line reply with no numbers, but it points the same way as this run's standalone result: on reranking, Jev is not the obvious win.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Search and tagging on keep.md
X: Jev reranks station suggestions in TrainLCD
YouTube: Jev in the reranking step of a RAG pipeline: why cosine similarity fails and how a steerable reranker compares. A Colab notebook comes with it.
X: LinkedIn bookmarks search
More Jev resources like this
A graded relevance evaluation of Jev as a reranker: 9,831 labelled pairs from 164 queries over a 33,047-item skills catalog, comparing Jev score reranks with BM25, bge-m3, and rank fusion.
Jev Practices & Patterns#community#python#search
Reranking benchmark that gave Jev, Cohere Rerank 4 Pro, ZeroEntropy zerank-2, and DeepSeek the same thirty BM25 candidates across eight English datasets, publishing saved responses, scoring code, and paired-bootstrap intervals. Jev's rubric scored 0.692 nDCG@10 against Cohere Pro's 0.691.
Jev Practices & Patterns#community#python#search
Community
anessbelbati/jev-rerank-benchPython
anessbelbati/jev-rerank-benchPythonYuchen Jin's take on Jev for LLM-as-a-judge work: 20-200x faster and 40-400x cheaper, with a screenshot of the model picking which candidate is AGI after a prompt sweep under $0.10.
Jev Practices & PatternsPost#community#x#evaluation
Community
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Jev Practices & Patterns#community#python#evaluation
Community
danielgshea/jev-as-a-judgePython
danielgshea/jev-as-a-judgePythonA probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonVendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.