Are the benchmark numbers real, or just launch material?

They recompute. Every raw API response is committed under cache/scores keyed by system, query and candidate-set hash, so all published figures can be regenerated from the shipped cache with no API key and no spend, and the analysis code carries 26 tests that make no API calls. The BM25 first stage is also checked against published Anserini anchors and warns on a miss.

If quality is a wash, should I pick Jev or the cross-encoder?

Decide on latency or cost. Jev is about 1.8x faster at the median and 2.4x at p95, and its worst case sits close to its typical case, which matters if the reranker runs inside a user-facing budget. Qwen3-Reranker-0.6B is 3.9x cheaper per query at roughly $0.00045 against $0.00178.

Does writing a smarter rubric actually help?

Not here. Two hand-written SciFact-specific rubrics both scored worse than plainly asking whether the passage is relevant to the query, and neither beat Qwen significantly. The README is careful that those rubrics were first drafts and were not selected on a held-out split, so it is not proof that rubric engineering never helps, only that the plain question is the cheapest thing to write and run.

Is the packing trick the real reason Jev looks fast?

Largely, and skipping it is a documented trap. TypeSafe's rerank cookbook issues one request per query-passage pair, which at depth 100 hits the documented 1,200 req/min ceiling at about 12 queries per minute. Putting the query in state and one question per passage in a single request gives identical scores with 1.56x fewer tokens; without it the 0.78s p95 would be roughly ten times larger.

Do I need an API key to check the results?

No. Every raw API response is committed under cache/scores, so the published numbers recompute from the shipped cache with no key and no spend; only re-running against the live APIs costs money.

What happens to oversized batches?

Requests over the 64k-token ceiling return HTTP 400, so the harness halves an oversized pack and retries. Questions are independent, so splitting a pack can never change a score.

How much does re-running the live benchmark cost?

scripts/run_all.sh all runs both datasets with repeats and rubrics for about $3.70, while scripts/run_all.sh nfcorpus runs one dataset for about $0.70, using list prices in a single editable table.

CommunityJev Practices & Patterns

Rerank Bench Jev

Benchmark of Jev as a production reranker against Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus. Ranking quality is a wash, Jev runs about twice as fast with a much tighter tail, and the cross-encoder costs roughly a quarter as much. Every raw API response is committed, so the numbers recompute without an API key.

See highlights

Open github.com

Stars: 1
Language: Python
License: APACHE-2.0
Published by: Community
Author: denser-org
Last verified: Oct 2026

denser-org/rerank-bench-jev

Organization repository on GitHub

View on GitHub

Production reranker benchmark: TypeSafe Jev vs Qwen3-Reranker-0.6B on BEIR SciFact and NFCorpus — nDCG@10, cost/query and p50/p95 latency. Quality is indistinguishable; Jev is ~2x faster, Qwen ~4x cheaper. From the team at denser.ai

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

Should you use TypeSafe's Jev — a "System One" model that returns calibrated probabilities instead of text — as a production reranker, or a dedicated cross-encoder?

Quoted from the project's README. Read the full README.

Tagscommunity awesomejev catalog-import

Works forJev for Classification Jev for Ranking Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Depth-100 rerank over a frozen BM25 first stage; both systems score all 100 passages in one request at the same concurrency.
The nDCG@10 gap is +0.022 on SciFact (p=0.011) and +0.005 on NFCorpus (p=0.53), and one SciFact repeat in three drops back below significance.
Latency is where they separate: Jev holds p95 at 0.76-0.78s against 1.69-2.10s, a tail only 0.12s above its own median instead of 0.6-0.9s.
Packing one question per passage into a single request returns identical scores (Pearson r = 1.000 at k=10) using 1 request instead of 100.

Watch out

Two public scientific datasets, both of which either model may have seen in training, and NFCorpus recall is structurally capped at 38.2 relevant documents per query against a depth-100 list. Cost is computed from list prices ($0.010/M on DeepInfra, $0.040/M on TypeSafe) in a single editable table, so it moves if either provider reprices, and re-running against live APIs costs about $3.70 for both datasets. The quality win does not replicate cleanly: one of three SciFact repeats fails to clear significance. The latency advantage is also partly an artifact of request packing, which the Qwen adapter matches by sending all 100 passages at once. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-rag-benchmark

Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

erendikmenn/jev-rag-benchmarkPython

jev-benchmark

Runnable benchmark putting Jev, jev-preview and Claude Sonnet 5 through one real agent task — classifying a tool call as readonly, destructive, privileged or exfiltration — on the same 60 labelled cases, with raw per-call results committed.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

themsquared/jev-benchmarkPython

minojev

Local decision models that return Jev-shaped typed distributions instead of generated text: freeze a backbone, cache candidate features once, train a roughly 0.8M-parameter decision head, calibrate per primitive.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

zeredy879/minojevPython

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

rorshopping/jev-on-a-laptopPython

local-jev

Local offline System One server speaking the hosted /v1/systemone wire format, so existing SDK code runs unchanged against small on-machine models.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

amithgc/local-jevPython

decision-index

Reproduction kit that rebuilds a frozen typed-decision benchmark suite from public sources and runs any engine through it with resume, scoring, and index computation.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

apolinario/decision-indexPython

From the Jev guides

Original write-ups on what this entry is for.

Designing the State: What to Include and What to Drop

Oct 12, 2026

Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.

Jev Practices & Patterns#state#design#primitives

Read article

Noul, End to End: One Probability and a Dozen Thresholds

Oct 12, 2026

A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.

Jev Practices & Patterns#noul#primitives#confidence

Read article

Scoring Without Measuring: Bands, Rubrics, and Composite Triage

Oct 12, 2026

A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.

Jev Practices & Patterns#score#primitives#scoring

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai