User repository on GitHub
300 public pages audited for AI citability with Jev decisions, checked against an LLM judge: agreement, cost and latency, measured
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev GEO Audit
A measured audit of 300 public pages for AI citability using nine Jev decisions per page, checked against an LLM judge: Jev agreed on 85.8% of 900 decisions and cost 158 times less per page, with latency measured for both.
- Stars
- 3
- Language
- Python
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- stas4000
- Last verified
- Oct 2026
- Languages
- Python
- Platforms
- OpenRouter
- Capabilities
- Open sourceRequires API key
Highlights
- Audits 300 pages from 10 public sites with one page-type question and eight yes/no checks each, turning probabilities into a score and a fix list.
- Cost was $0.0228 for 300 pages versus $1.2063 for the judge on 100 pages: $0.000076 versus $0.012063 per page, 158 times more for the judge.
- Median latency per page was 61.1 ms for Jev across 75 batched requests versus 2,837 ms for the LLM judge.
- Jev and the judge agreed on 85.8% of 900 decisions, and on 98.2% when Jev's own confidence was 0.90 or higher.
- 122 of the 128 disagreements are decisions where Jev itself was under 0.90, so confidence is a usable signal for escalation to a bigger model.
Quickstart
bash
export OPENROUTER_API_KEY=...
python3 fetch.py
python3 audit.pyWatch out
Grounded in the truncated repository README and GitHub metadata reviewed on 2026-10-15; the README's run instructions were cut off after the audit.py line. The judge is another model, not ground truth, so 85.8% is agreement rather than accuracy, and the sample covers 10 selected public sites.
Questions about this entry
Short answers drawn from the linked project or page.
Whether an AI assistant would quote a page, run as nine Jev decisions per page: one page-type classification and eight yes/no citability checks, turned into a per-page score and a fix list, with GEO standing for generative engine optimization.
The same nine checks were put to anthropic/claude-opus-5 on a seeded sample of 100 pages. The two agreed on 85.8% of 900 decisions: 85.5% of the 800 yes/no checks and 88% of the 100 page-type labels.
Jev cost $0.0228 for 300 pages, $0.000076 per page and 61.1 ms median per page. The judge cost $1.2063 for 100 pages, $0.012063 per page and 2,837 ms median, 158 times more per page.
When Jev's confidence in its own answer was 0.90 or higher, 324 of the 900 decisions, it agreed with the judge 98.2% of the time; below 0.90 agreement fell to 78.8%, and 122 of the 128 disagreements sat in that low-confidence region.
Contains concrete numbers or dated facts agreed on 81%, followed by first paragraph answers the title and compares options at 82% and states limits at 83%. Some disagreement has a direction: Jev was stricter on titles and limits, and more generous on self-contained claims.
No. The README states that the judge is another model, not ground truth, so 85.8% is agreement with that judge rather than accuracy. Rolled into citability scores the means are close (71.7 for Jev, 73.8 for the judge) but noisy page by page.
You need Python 3.10 or newer and an OpenRouter API key with access to ~typesafe/jev-latest; no packages to install. Run fetch.py to collect the pages, audit.py to score them with Jev, and eval.py to compare against the judge.
More Jev resources like this
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonBenchmark comparing TypeSafe Jev 1.13 with the local Laya MLX typed-decision model across 1,470 synthetic items and eight tasks: accuracy, calibration, latency and throughput, with scripts, datasets, predictions and the full report in the repo.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
harrymunro/jev-laya-benchmarkPython
harrymunro/jev-laya-benchmarkPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonIndependent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonClassifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision each, then checks those labels against an LLM judge on a seeded sample of 100. Ships three scripts and a static topic-map page.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Communitystas4000/jev-papersPython
stas4000/jev-papersPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.