User repository on GitHub
1,000 arXiv AI papers classified with one Jev decision each, checked against an LLM judge. Open rebuild, MIT.
- Forks
- 3
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev Papers
Classifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision each, then checks those labels against an LLM judge on a seeded sample of 100. Ships three scripts and a static topic-map page.
- Stars
- 7
- Language
- Python
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- stas4000
- Last verified
- Oct 2026
Used inJev for Data & Analytics
- Languages
- Python
- Platforms
- OpenRouter
- Capabilities
- Open sourceRequires API key
Highlights
- 1,000 papers from cs.AI, cs.LG, cs.CL and cs.CV (2026-09-16 to 09-18) classified into 24 topics in one committed run.
- Total Jev cost for the run: $0.0585, or $0.000058 per paper; the Opus-class judge cost $0.8940 for its 100 papers.
- Latency per paper: 56.9 ms median (90.5 ms p95) for Jev; 1,923 ms median (2,810 ms p95) for the judge.
- Agreement on the 100 sampled papers: 85 same topic; 98% agreement when Jev confidence was 0.90 or higher.
- Three scripts (fetch.py, classify.py, eval.py) and a static site/index.html topic map; MIT licensed.
Quickstart
bash
export OPENROUTER_API_KEY=...
python3 fetch.py
python3 classify.py
python3 eval.pyWatch out
Copy reviewed against the repo README on 2026-10-15; the README states plainly that the judge is another model, not ground truth, so 85% is agreement rather than accuracy, and all figures come from the single committed run under data/.
Questions about this entry
Short answers drawn from the linked project or page.
fetch.py pulls 1,000 recent arXiv AI papers from the public API; classify.py asks Jev one 24-way Choice question per paper, four papers per request with six requests in flight; eval.py asks a strong chat judge the same question on a seeded sample of 100 and reports disagreements; site/index.html renders the result as a topic map.
The committed run totals $0.0585 for 1,000 papers with jev-1.13-20260917, about $0.000058 per paper, against $0.8940 for the judge's 100 papers. The README's commands are: export OPENROUTER_API_KEY, then `python3 classify.py` (about 6 cents) and `python3 eval.py` (about 90 cents).
On the 100 sampled papers: 85 same topic; the judge's pick was Jev's first or second choice on 95; when Jev's confidence was 0.90 or higher (62 papers) they agreed 98%, versus 63% below 0.90. 14 of the 15 disagreements were papers where Jev itself was under 0.90, so confidence flags which labels to double-check.
No, and the README says so plainly: the judge is another model, not ground truth, so 85% is agreement, not accuracy. It also notes no disagreement pair repeats, with the judge leaning toward theory (4 of 15) and agents (3 of 15).
The 24 topics live in topics.json with one line of criterion each; edit them and run classify.py --fresh for your own map. eval.py --judge <model id> accepts any OpenRouter chat model, and --seed or --sample change the sample. classify.py and eval.py are resumable, picking up only what is missing after a failed request.
Tile area is the paper count per topic; clicking a topic lists its papers, everything is searchable, and every paper shows Jev's confidence. It reads site/data.js, which the scripts rewrite, so it works from disk with no server.
Python 3.10 or newer and an OpenRouter API key with access to ~typesafe/jev-latest; no packages to install. fetch.py is a no-op on a second run unless you pass --refresh, and it takes under a minute because arXiv asks for three seconds between requests.
More Jev resources like this
A measured audit of 300 public pages for AI citability using nine Jev decisions per page, checked against an LLM judge: Jev agreed on 85.8% of 900 decisions and cost 158 times less per page, with latency measured for both.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Communitystas4000/jev-geo-auditPython
stas4000/jev-geo-auditPythonBenchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonA pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades against nano-class and frontier LLMs and a supervised encoder, plus a second study of confidence under prompt injection.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
ickma2311/jev-baselines-evalPython
ickma2311/jev-baselines-evalPythonA Python and JavaScript package that loads any Hugging Face dataset, auto-adapts evaluation rubrics to the domain, and classifies every row through a single parallel Jev call with calibrated probabilities, plus a CLI and an interactive studio for uploads and export.
Jev Repos & SDKs#community#catalog-import#logicrw
Community
hemanth/hfjevTypeScript
hemanth/hfjevTypeScriptA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.