Jev directory
CommunityJev Benchmarks & Evaluations
Jev Lab — SuperInstance
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is Jev Lab — jammaru.
Highlights
- Baseline separation between true and false claims measured at 0.9800, with 0.000 spread on both.
- An empty state scored 0.99 on a world fact, the same as a correctly read claim.
- With two indistinguishable options, Jev picked the first one 9 times out of 9 at P = 0.86.
- Semantic triage over 24 fleet READMEs reached precision 0.909 and recall 0.909.
- On a self-contradictory state the score spread was 0.040, twice the house noise band.
Questions about this entry
Short answers drawn from the linked project or page.
Every number in the lab was measured against api.typesafe.ai/v1/systemone using model jev-latest, on 2026-10-18, with repeats and the spread kept next to each verdict.
The repo commits the underlying data and an analyze-e7.py script that recomputes the E7 arrangements. The recomputation corrected the README's earlier claim of 9/9 in every arrangement to 6 of 7.
An empty state scored 0.99 on a world fact, the same score as a correctly read claim. The house capability map recorded this at 0.610, so a harness whose state fetch fails can now report perfect confidence.
Given two indistinguishable options, Jev picked the first one 9 times out of 9 at P = 0.86. That retracted E1b's BARE arm, which had been measuring a position preference rather than a reading.
No. In E2b it scored the two directly stated clauses correctly but missed the inferential clause, which required composing two clauses into a rule about a different record.
Ranking 24 real fleet READMEs against a grep-verifiable ground truth gave precision 0.909 and recall 0.909. In two of the three disagreements, Jev was right and grep was the artefact.
It is published under the SuperInstance GitHub organization as SuperInstance/jev-lab, with 0 stars at the time of listing. The provided material does not state a license.
More Jev resources like this
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonIndependent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
A timestamped benchmark and trading arena that replays resolved prediction markets: models commit to probabilities using only information available at the time, and realized P&L is the headline metric with Brier, log-loss and ECE as checks.
Jev Benchmarks & Evaluations#community#github#python
Community
ruyianry/JevGymPython
ruyianry/JevGymPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.