User repository on GitHub
Test harness + benchmark for TypeSafe's Jev decision model (noul/choice/score) via OpenRouter's Decisions API
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Jev Test
Small test harness and benchmark for Jev through OpenRouter's Decisions API. It ships a thin decide wrapper, 17 documentation-derived cases with 23 checks covering Noul, Choice and Score, and a script comparing Jev against GPT-5.6 Terra and Claude Opus 5.
- Stars
- 0
- Language
- Python
- Published by
- Community
- Last verified
- Oct 2026
About this project
A small test harness for Jev (typesafe/jev-latest), TypeSafe's structured decision model, called through OpenRouter's alpha Decisions API (`POST. Jev doesn't generate text. You send it a state (the text or JSON to judge) and a set of typed questions. It returns typed answers with calibrated probabilities, and your code decides what to do with them. .env is git-ignored. Never commit it.
Quoted from the project's README. Read the full README.
Works forJev for Evaluation
Used inJev for Software & DevTools
- Languages
- Python
- Platforms
- OpenRouter
- Capabilities
- Open sourceRequires API key
Highlights
- All 23 checks passed across 17 cases for about $0.0003 in the recorded 2026-09-19 run.
- Median latency measured: Jev 452 ms, GPT-5.6 Terra 1,505 ms, Claude Opus 5 3,394 ms.
- Total cost across the 17 calls: $0.00027 for Jev versus $0.00971 and $0.05223 for the LLM baselines.
- Runs with uv on Python 3.12; cases are taken from TypeSafe and OpenRouter documentation.
Watch out
No license file or SPDX license is present. The benchmark is a single small run, and the latency figures include network time from the author's machine.
Questions about this entry
Short answers drawn from the linked project or page.
Accuracy against documented expectations, median and p90 latency, token counts and cost, running the same cases on Jev and on general LLMs that are graded with identical checks.
They get the same state, instructions and criteria through OpenRouter chat completions with a strict JSON schema, returning a probability, a label or a level index that the shared checks grade.
All 23 checks across 17 cases passed for about $0.0003 in total, with median latencies of 452 ms for Jev, 1,505 ms for GPT-5.6 Terra and 3,394 ms for Claude Opus 5. Total cost over the 17 calls was $0.00027 for Jev against $0.00971 and $0.05223. The README calls this a single small run.
No, and the README says so plainly: all three models scored 23 of 23, so on these clear-cut cases the difference is speed and cost, not correctness. Token counts are also not comparable, since Jev's input count includes its own prompt scaffolding, so the README points you at cost instead.
The docs' Safari bug scored 1.51 at confidence 0.26, which is an escalate, using plain tier strings. With what and examples tiers the same case scored 1.02 at confidence 0.97, an act. The quickstart ticket still routed to billing but at 0.80, inside the confirm band rather than act, with a 13% chance of technical.
Install uv, run uv sync for Python 3.12 with requests and python-dotenv, then put OPENROUTER_API_KEY in .env. jev.py is a thin wrapper around the Decisions endpoint. Jev bills input tokens only at $0.042 per million with output free, which is why 17 calls came to $0.00027. There is no license file or SPDX identifier in the repository.
Every answer is labelled act above 0.9, confirm from 0.5 to 0.9, and escalate below 0.5, and the README's advice is to raise the thresholds as the risk goes up, since reading a balance needs less confidence than approving a transfer. For Score, the value is the probability-weighted mean of level indexes and can land between integers, so probabilities carry the full picture.
More Jev resources like this
0GitHub stars
Reproducible billing and stability measurement of Jev's fan-out behavior across 2,976 requests to jev-1.13-20260917: asking N questions in one call bills the state once, with raw data, exact invoice checks and a batched-versus-single answer comparison.
Jev Benchmarks & Evaluations#community#benchmarking#billing
Community
blowxian/jev-fanout-benchPython
blowxian/jev-fanout-benchPython2GitHub stars
Pre-registered benchmark adding Jev 1.13 to Stanford MedHELM's MedHallu results: one Noul asks whether a proposed answer misrepresents its PubMed abstract. Jev scored 92.9% on 1,000 test items at a 204 ms median and USD 0.03 per 1,000 checks through OpenRouter.
Jev Benchmarks & Evaluations#community#medical#hallucination
Community
stperic/jev-medhallu-benchmarkPython
stperic/jev-medhallu-benchmarkPython2GitHub stars
Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.
Jev Benchmarks & Evaluations#community#benchmark#calibration
Community
jjd-lab/jev-synthetic-surveyPython
jjd-lab/jev-synthetic-surveyPython15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython7GitHub stars
Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter Notebook3GitHub stars
A benchmark that steers decision models through 100 key-and-door grid puzzles with one Jev Choice per move, scored against an exact BFS solver; it found Jev picks an optimal move about 90% of the time yet finishes only 57 of 100 games.
Jev Benchmarks & Evaluations#community#benchmark#gridworld
From the Jev community
Posts from builders shipping with Jev right now.