Docs
OfficialJev Cookbooks
Cookbook: Re-ranking
One question per query-candidate pair raises top-1 accuracy from 5% to 18% and top-10 accuracy from 38% to 62% on 40 CLERC legal queries.
- Format
- Docs
- Reading time
- 2 min
- Published by
- TypeSafe AI
- Last verified
- Sep 2026
Used inJev for Legal & Compliance
Highlights
- BM25 shortlists 30 candidates per query over 3,565 CLERC passages; the gold passage appears in the shortlist 100% of the time but at rank 1 only 5%.
- One Noul per query-candidate pair (1,200 calls across 12 threads) lifts top-1 from 5% to 18% and top-10 from 38% to 62%.
- The 1,200 calls used 1,536,002 input tokens and 25,200 output tokens, costing $0.0645 at $0.042 per 1M input tokens.
- The scoring question asks whether the candidate could come from the cited precedent, with NoulCriteria spelling out true and false cases.
Quickstart
python
question = Noul(
instructions="Is this candidate the cited case?",
criteria=NoulCriteria(
true="The candidate states the specific rule the query cites.",
false="The candidate is only on a similar topic.",
),
)
response = client.system_one(
state={"query_excerpt": query, "candidate_passage": candidate},
questions={"is_cited_source": question},
)
noul = response.answers["is_cited_source"].noul # -> 0.87Watch out
Re-ranking can only reorder the shortlist fast search already produced — it cannot recover a passage BM25 failed to select.
Questions about this entry
Short answers drawn from the linked project or page.
It lifted top-1 from 5% to 18% and top-10 from 38% to 62% over 40 CLERC legal queries, with one Noul per query-candidate pair — 1,200 calls using 1,536,002 input tokens and costing $0.0645 at $0.042 per million.
It can only reorder the shortlist fast search produced. BM25 surfaced the gold passage in the shortlist 100% of the time but at rank 1 only 5%, and re-ranking cannot recover a passage BM25 failed to select at all.
3,565 court opinion passages from the CLERC dataset. BM25 builds a shortlist of 30 candidates for each of 40 queries, and each shortlist is then scored and reordered. The gain comes from putting the right passage first, not from finding it.
A plain yes or no cannot rank 30 candidates, and inventing a numeric scale means applying the same standard consistently across every pair, which repeated calls by a general-purpose model do not guarantee. A Noul returns a number between 0 and 1 for the same question every time, and that is the score you sort on.
The NoulCriteria spell out both sides: true means the candidate states the specific rule the query cites, false means it is only on a similar topic. Those criteria are applied to every query-candidate pair, so the score means the same thing across the shortlist.
1,200 calls across 12 threads, using 1,536,002 input tokens and 25,200 output tokens for $0.0645 at $0.042 per million input tokens. Every ask is one query paired with one candidate passage, so cost tracks shortlist size times query count rather than corpus size.
To keep the attention on re-ranking, since re-ranking only ever sees the passages that make the shortlist and the choice of fast search method is a side issue. BM25 ranks passages by shared words, and real systems often combine keyword search with dense embeddings; the cookbook tests BM25 alone.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Search and tagging on keep.md
X: Jev reranks station suggestions in TrainLCD
YouTube: Jev in the reranking step of a RAG pipeline: why cosine similarity fails and how a steerable reranker compares. A Colab notebook comes with it.
X: LinkedIn bookmarks search
More Jev resources like this
Score retrieved passages with four Noul questions each, then route in code: prompt injections dropped, false premises kept as conflicts, and evidence assembled separately for the generator.
Jev CookbooksDocs#official#cookbook#search
Official
Rank 182 agent skills in one request and re-read the top three in a second: over 488 requests, wrong skill loads fell from 16.8% to 7.3% and needless loads from 9.8% to 4.0%.
Jev CookbooksDocs#official#cookbook#agents
Official
A 13-question regulatory briefing that shows batching every question into one call is 12.2x cheaper and 10x faster with no change in answers.
Jev CookbooksDocs#official#cookbook#batching
Official
Semantic search over GitHub's Terms of Service: one request ranks all 218 lines with a Choice while a Noul checks whether the document contains an answer at all, including when it should say no.
Jev CookbooksDocs#official#cookbook#search
Official
Decide which of 450 candidate product pairs from two beer catalogues describe the same entity: one three-level Score per pair splits 40 merges, 50 curator reviews, and 360 non-matches.
Jev CookbooksDocs#official#cookbook#entity-matching
Official
A repeatability study: an 8-question moderation rubric run 15 times per condition shows a mean probability standard deviation of 0.0098, and a 0.60 uncertainty gate lifts agreement to 99.2%.
Jev CookbooksDocs#official#cookbook#consistency
Official
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.