Jev RAG Benchmark

No. The premise is that Jev can improve, stay flat, or get worse on quality, latency, and cost. The repo keeps full tables, confidence intervals, and ablations that reject some Jev uses such as first-stage retrieval.

All rerankers get the same frozen top-20 hybrid candidates and generation is disabled in that stage, so only passage selection is measured. Generation is compared separately on identical frozen Jev top-five contexts.

No. Retrieval recall, ranking scores, and answer F1 measure different stages. References are short extractive answers while generators emit sourced sentences, so exact match stays near zero even when ranking is strong.

macOS or Linux with Python 3.11 to 3.13, uv, and an OpenRouter key. Published model and embedding calls use OpenRouter with no local LLM, and doctor plus smoke commands check setup without paid calls.

On the same frozen hybrid top-20 across 1,044 XQuAD-TR questions, Jev 1.13 batch Noul and Cohere Rerank 3.5 recover an identical 1,038 gold passages into top-5. Jev costs 60.6 percent less, $0.410866 against $1.044 for all 1,044 questions, while Cohere leads on ranking by 0.529 nDCG points and by 66 ms at the median. Against no reranker at all, Jev adds 19 top-5 recoveries and 3.783 nDCG points with zero fallbacks.

Two, clearly. Pointwise Jev recovered nothing and cost 80.8 percent more latency and 56.5 percent more cost. Full-corpus hierarchical Jev was rejected outright as a retriever replacement, at 76 percent Recall@5 against plain BM25 top-5 at 97 percent. A multi-signal evidence router over-filtered into 10 fallbacks and is kept only as a security mode that has to be recalibrated first.

With both generators handed the exact same frozen Jev top-5 contexts, DeepSeek V4.1 Flash beat Gemini 3.8 Flash: 194 successful answers against 163, mean F1 29.191 against 27.274 percent, valid citation IDs 95.69 against 93.87, and end-to-end cost 62.7 percent lower. The trade is latency, a 2.44 times slower median and 3.55 times worse p95. Neither is anywhere near answering every question correctly.