User repository on GitHub
Benchmark TypeSafe Jev against any OpenRouter model on your own data.
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev Eval — 4esv
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
- Stars
- 1
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- 4esv
- Last verified
- Oct 2026
- Languages
- Python
- Platforms
- OpenRouterCLI
- Capabilities
- Open sourceRequires API keyRuns locally
Highlights
- Five tasks with 300 items each: CLINC150 intent, Banking77 intent, SST-5 sentiment, IMDB polarity as noul and IMDB polarity as choice.
- On out-of-sample CLINC150 (151 options) Jev scores 0.897 versus 0.610 for the 435M open-jev; on Banking77 open-jev leads 0.873 to 0.780.
- Only Jev stays flat with option count: p50 latency moves 0.17 s at 2 options to 0.20 s at 151 options; local models degrade roughly 4-7x.
- Local checkpoints Laya (421M), open-jev (435M) and Kev-0.8B run on an M-series GPU via uv sync --extra local; Kev already serves the /v1/systemone contract.
- The README warns that training data decides the winner: open-jev lists Banking77 and SST-5 in its training data, so quoting one number alone reverses the story.
Quickstart
bash
cp .env.example .env
uv sync
uv run python -m evaljev.run --model jev
uv run python -m evaljev.reportWatch out
Copy reviewed against the truncated repo README on 2026-10-15; the excerpt stops inside the Kev server instructions. All figures are the project's own benchmark results, GitHub metadata lists no license, and other jev-eval repos from different authors already exist in this catalog.
Questions about this entry
Short answers drawn from the linked project or page.
It benchmarks TypeSafe Jev against any OpenRouter model or a local checkpoint on labelled classification data, reporting accuracy, calibration (ECE), confidence distribution, p50 latency and cost across five shipped tasks.
On 300 items per task: 0.897 on out-of-sample CLINC150 (151 options), 0.780 on Banking77 intent, 0.570 on SST-5 sentiment, 0.970 on IMDB polarity as noul and 0.967 as choice, per the README's tables.
ECE shows whether confidence means what it says; Jev records 0.039 on CLINC, while the README notes open-jev is the best-calibrated model on four of five tasks at 435M parameters and no cost.
Copy .env.example to .env with TYPESAFE_API_KEY and OPENROUTER_API_KEY, run uv sync, then uv run python -m evaljev.run --model jev, or pass an OpenRouter model name. uv run python -m evaljev.report renders the results.
Yes: uv sync --extra local installs the local extra, then you run models like laya, open-jev or Kev. Any server speaking the System One contract is benchable by URL, and the README shows the HF_HUB_DISABLE_XET=1 prefix for the shipped checkpoints.
Running with --tag rerun executes a second tagged run to check determinism; the README includes it among the standard commands alongside the pytest suite.
The README warns that open-jev and Kev list Banking77 and SST-5 in their training data and beat Jev there, while on CLINC150, the same kind of task but absent from their training lists, they trail by 28.7 points. Quote either number alone and the story reverses.
Other listings of this project
Forks and copies of this project in the directory. This entry is the canonical one.
Third-party TypeScript check that compares Jev (typesafe-ai/jev) against openai/gpt-4o-mini and anthropic/claude-sonnet-4.5 under identical conditions, routing 60 synthetic booking inquiries in four languages for a photo-shoot service, all through one Vercel AI Gateway key.
Jev Benchmarks & Evaluations#community#benchmark#comparison
Community
Shogo-nfrealmusic/jev-evalTypeScript
Shogo-nfrealmusic/jev-evalTypeScriptA measurement study that scores Jev against gpt-5.4-mini and gpt-5.6-luna on four public classification sets, then against recorded outcomes on a few thousand real pipeline decisions. Every per-item answer and the scoring code are committed.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
onlyoneaman/jev-evalTypeScript
onlyoneaman/jev-evalTypeScriptMore Jev resources like this
A measured audit of 300 public pages for AI citability using nine Jev decisions per page, checked against an LLM judge: Jev agreed on 85.8% of 900 decisions and cost 158 times less per page, with latency measured for both.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-geo-auditPython
stas4000/jev-geo-auditPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonIndependent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonA reproducible lab measuring how several coding agents coordinate on one repository, plus Médula, a kernel that asks TypeSafe Jev before every write whether a change collides with what the other agents are doing.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
joaquinruiz/medulaPython
joaquinruiz/medulaPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonBenchmark comparing TypeSafe Jev 1.13 with the local Laya MLX typed-decision model across 1,470 synthetic items and eight tasks: accuracy, calibration, latency and throughput, with scripts, datasets, predictions and the full report in the repo.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
harrymunro/jev-laya-benchmarkPython
harrymunro/jev-laya-benchmarkPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.