Docs
CommunityJev Benchmarks & Evaluations
Jev RAG Chinese Benchmark (S1)
cnfeichen's reproducible Chinese reranking benchmark for Jev over hard negatives from C-MTEB/T2Reranking: a 4-level Jev Score rubric reaches 0.546 nDCG@5 on 100 queries against 0.413 for BM25 with jieba. Independent community research, not affiliated with TypeSafe. MIT.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- cnfeichen
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- Jev Score (4-level rubric, Chinese) reaches 0.546 nDCG@5 [0.481, 0.611] and 0.602 nDCG@10 over 100 queries.
- BM25 with jieba reaches 0.413 nDCG@5; the paired delta for Jev Score minus BM25 is +0.133 [+0.051, +0.232].
- Both Jev variants beat BM25 at R@5, 0.830 against 0.690; Jev Noul is the binary variant.
- The single run cost $0.07 against the pinned typesafe/jev-1.13-20260917 model.
- Ships s1.jsonl and dry.jsonl pools, every API call log, results with bootstrap CIs and a Noul threshold sweep.
Watch out
Card reviewed 2026-10-17; the headline numbers are the author's own single run per variant over 100 queries, so the confidence intervals are query-level and not multi-run evidence. The pools sample C-MTEB/T2Reranking content, and the card states the work is independent community research not affiliated with TypeSafe.
Questions about this entry
Short answers drawn from the linked project or page.
Reproducible Chinese-language reranking measurements for Jev (TypeSafe System One), run over hard negatives from C-MTEB/T2Reranking. It is published as an S1 snapshot of 100 queries with one positive and nine hard negatives each, plus logs and a frozen protocol.
Jev Score with a 4-level rubric reaches 0.546 nDCG@5 [0.481, 0.611] and 0.602 nDCG@10; Jev Noul (binary) reaches 0.531 and 0.588; BM25 with jieba reaches 0.413 and 0.513. Jev Score's paired delta over BM25 is +0.133 [+0.051, +0.232], excluding zero.
Noul asks a binary yes/no relevance question, while Score applies a 4-level relevance rubric; both are Chinese-language variants run against the pinned typesafe/jev-1.13-20260917 model, and both beat BM25 at R@5 with 0.830 against 0.690.
100 queries sampled with seed 42, each with one positive and nine hard negatives from T2Ranking lineage; one run per variant; bootstrap confidence intervals over queries; total API cost $0.07. The full report adds a Noul threshold sweep, and every API call is logged.
s1.jsonl and dry.jsonl sampled pools, a logs directory with every API call (timestamp, variant, latency, frozen-question hash, returned model id, answer, usage and cost), results with confidence intervals, and PROTOCOL.md.
No: the card describes independent community research not affiliated with TypeSafe AI. It is a single run per variant with limitations documented in the protocol and the report, and the code and scripts live in a separate GitHub repository.
MIT, per the repository metadata, with the language tagged as Chinese only. The pools sample content from C-MTEB/T2Reranking, so check that source dataset's terms for the underlying text before redistribution.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Search and tagging on keep.md
X: Jev reranks station suggestions in TrainLCD
YouTube: Jev in the reranking step of a RAG pipeline: why cosine similarity fails and how a steerable reranker compares. A Colab notebook comes with it.
More Jev resources like this
897,035 search-agent decision rows in unified JEV format across train, validation and test, covering retrieval judgment, query understanding, evidence judgment and search actions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPythonRun outputs behind the JevGPT paper's tables: 830 BrowseComp-Plus records at 90.48% accuracy and 105 NeuCLIRBench topics at 0.5390 nDCG@20, produced by a GPT-5.5 System Two paired with Jev-1.13.0 as System One. Per-item cost, latency and encrypted trajectory files are included. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.