Jev directory
CommunityJev Benchmarks & Evaluations
Jev Localization Bench
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
- Stars
- 1
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- 200 real issues from SWE-bench Verified, scored against the human fix that was actually merged.
- Sonnet leads overall at 76.0% hit@1; Jev's fan-out configuration reaches 69.0% at 83x lower cost.
- On the 81 bugs where Jev's confidence is 0.8-1.0, both models score 87.7% and pick the same file on 78 of 81.
- A 0.80 confidence gate matches Sonnet's hit@1 for 1.6x less money and 1.6x less latency.
- Sonnet returned 36 file paths outside the candidate list across 200 runs; Jev's count is 0 by construction.
Questions about this entry
Short answers drawn from the linked project or page.
One decision a coding agent makes on every task: given a bug report, which file needs editing? It uses 200 real issues from SWE-bench Verified where the correct answer is known from the human fix that was merged, and every method sees identical inputs.
Sonnet wins overall at 76.0% hit@1. Jev scores 70.0% with one call per file, 69.0% with one fan-out call, and 71.0% with LLM-compiled questions. The aggregate gap is reported as real, with bootstrap confidence intervals and McNemar tests on the disagreements.
On the 81 bugs where Jev's confidence is 0.8-1.0, about 40% of the workload, both models score 87.7% and pick the same file on 78 of those 81, with zero McNemar disagreements. Below that band Sonnet wins every band.
Jev costs $0.001594 with one call per file, $0.000544 with one fan-out call, and $0.002024 with LLM-compiled questions. Sonnet costs $0.0453. Every cost comes from the API's own usage field and every latency is wall clock.
At a 0.80 threshold, 60% of instances escalate, hit@1 is 76.0%, cost is $0.0280 per instance, and mean latency is 2.31s. Escalated instances are charged for both calls, so savings cap near 1.6x however cheap the first stage is.
The jev_compiled configuration asks Sonnet once to design the Jev program from a task description, then runs that program. It scored 71.0% against the hand-written 70.0%, but cost 29% more per instance because it emitted four questions where two feed the ranking.
Localization is one step of an agent loop, so it does not measure solve rate. The BM25 prefilter caps everyone and dropped the answer on 28 of 200 instances. Only Jev was gated, and the study covers one benchmark, Python only, 200 issues.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
X: A group chat bot with decision room routing
More Jev resources like this
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonBenchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonA sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.