Question 1

Are the benchmark numbers real or just copied tables?

Accepted Answer

Jev and local monoBERT rows are measured runs with frozen manifests, timing, and usage retained; paper rows from TREC overviews are labeled historical references, not reproductions or paired tests.

Question 2

Why does Jev win MAP yet lose nDCG@10 on passages?

Accepted Answer

Against local monoBERT on castorini/monobert-large-msmarco, Jev original-text and matched-window runs improve MAP while scoring lower on nDCG@10, which uses all grades while MAP counts only grades 2-3 as relevant.

Question 3

What is the difference between matched windows and full documents?

Accepted Answer

Matched windows feed Jev the same decoded windows as monoBERT to control for input; full-document scoring lets Jev see title, headings, and body at once with MaxP taking each document highest window score.

Question 4

Can I explore results without the licensed WSJ text?

Accepted Answer

Yes: the simulator replays the fixed WSJ lexical ranking and completed Jev scores from seeded Neon tables, displaying the top 10 while article bodies stay outside Git and the database.

Question 5

What do these runs cost in API time and dollars?

Accepted Answer

Estimated API cost per complete run, with the per-query median alongside. On MS MARCO passages, JEV spent about $0.76 at a 37 to 38 second query median, against a local monoBERT at 22.46 seconds and $0 hosted API. On MS MARCO documents, large-window MaxP spent $1.762336 at 4.81 seconds. On WSJ, complete-document JEV was the cheapest and fastest at $0.328312 and 4.02 seconds, against passage MaxP at $0.605227 and 14.98 seconds and monoBERT at 30.27 seconds. Local compute is not included.

Question 6

How should I read the two different WSJ MAP numbers?

Accepted Answer

They measure different scopes and the README separates them deliberately. The web app's WSJ view evaluates with a rank-100 cutoff, trec_eval -c -M100, where nDCG@10 goes 0.4640 to 0.6634 and MAP@100 goes 0.1729 to 0.2263. The WSJ table reports full 1,000-result MAP of 0.2521 to 0.3055. In both cases only positions 1 to 100 are reranked; positions 101 to 1,000 keep their BM25 order, so the reranker is judged on the head of the ranking rather than the whole list.

Question 7

Where does the code and the data come from, and what licence applies?

Accepted Answer

The experiment workflow draws on karpathy/autoresearch under MIT, and the WSJ retrieval engine derives from andrewtrotman/JASSjr under BSD-2-Clause. The project itself is MIT licensed except for that upstream-derived code, which is called out in THIRD_PARTY_NOTICES.txt. Dataset text, model downloads, credentials and response caches stay outside Git, the stage-1 baseline is fixed read-only history that changes on main do not replace, and WSJ article bodies stay outside both Git and the simulator database.

Jev Reranking

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community