Question 1

So is a semantic sort safe to ship, or not?

Accepted Answer

Both, depending on the task. On 360 human-labeled rows jev-1.13.0 passed all six pre-registered gates, with boolean inversion 0.036 and Score ordinal inversion 0.143 against a 0.15 threshold. On Amazon ESCI's 306 graded pairs over 30 hard shopping queries it failed four of six, with boolean inversion 0.255 and 23 of 30 queries over the line. The Score figure is also the weak link, and it passes on a coarse 3-level proxy rather than real relevance grades.

Question 2

Who ran these numbers, and should I trust the author?

Accepted Answer

It is independent of TypeSafe, but the author admits that commitjev, jobbyjev and jevq are also his, so read those as applications of these results rather than corroboration. The vendor's only published figure is 67.8% agreement against averaged frontier judgments, self-run and unreproduced. His own instruction is the fair one: disagree by re-running the harness rather than taking anyone's word for it, including his.

Question 3

ORDER BY prob DESC LIMIT 20 — is that even a ranking?

Accepted Answer

Not reliably. Probabilities come back at two decimals: 360 rows produced 45 distinct values, 21.6% of row pairs tie exactly, and 53 rows sit at 0.99. SQL does not define the order of tied rows, so LIMIT 10, LIMIT 20 and LIMIT 50 all cut inside that 53-way tie and return an engine-dependent sample. The inversion rate cannot see it, since tied rows mostly share a label. Mitigations: treat LIMIT as a filter, break ties with a second question or Score confidence, or at minimum add a deterministic secondary key.

Question 4

Does batching rows to save tokens change the results?

Accepted Answer

It does, which is the finding most extensions will trip on. The same rows sent through one integration's default 40-row batching failed the ranking gate at inversion 0.171 against the 0.15 threshold that one row per request passed. That is why the author's own tools send one item per request and pay the extra calls. Reproduce with harness/run_calibration.py, about $0.013 for the full run, and --analyze-only replays cached responses for free.

Question 5

Does a prob > 0.9 filter behave the way it reads?

Accepted Answer

It is stricter than it reads. Eight of ten reliability bins sit above the diagonal with a mean signed gap of +0.042, and the measured true rate at a predicted 0.855 is 0.914, so calibrate against the published table rather than intuition. Per-probe jev_bool ECE runs 0.030 medical, 0.042 forsale and 0.062 space.

Question 6

Which tools does this measurement actually apply to?

Accepted Answer

Every extension built on the same API. Three DuckDB integrations shipped in the release week, colliber/duckdb-jev, recodelabs/duckdb-jev and Query-farm/vgi-typesafe, plus pg-jev for Postgres, and all of them expose ORDER BY over a Jev probability without shipping a measurement of it. The two-decimal tie behaviour is a property of the output, so it applies identically to each.

Question 7

Can I reuse the corpus, or only the harness?

Accepted Answer

Only the harness. MIT covers the code, not the corpus and not Jev's outputs, which TypeSafe's terms govern, and the ESCI probe carries one seed and 30 queries while 20 Newsgroups has no explicit license. Fresh scoring runs need Python and a TypeSafe key, but the corpus, client, metrics, gate and notebook all run without one.

Jev Orderby Bench

About this project

Highlights

Quickstart

Questions about this entry

From the Jev guides

From the Jev AI model community