Jev directory
CommunityJev Benchmarks & Evaluations
Jev Study
Portuguese-language technical study of Jev, TypeSafe AI's System One decision model, benchmarked against a structured-output LLM baseline. Ships a Python 3.11 harness, a 45-task synthetic eval set, and a runner that records decisions, confidence, latency, tokens, and estimated cost.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Python 3.11+ project using uv, Ruff, and pytest; CI runs lint and tests without TypeSafe or LLM keys.
- Gabarito v1 holds 45 synthetic tasks, 15 each of Choice, Score, and Noul; six items are written as low_confidence.
- JevHttpBackend calls POST /v1/systemone through the standard library and refuses redirects so the Authorization header stays on host.
- Runner records backend, requested and responding model, decision, confidence, probability, latency, tokens, and estimated cost.
- No model results are published in the repository; the calibration claim stays a hypothesis until a live run.
Questions about this entry
Short answers drawn from the linked project or page.
The project uses uv. Running uv run jev-study executes the fifteen placeholder tasks, while the evaluate-jev command reads the v1 gabarito in data/eval/decisions.v1.jsonl and runs either the echo backend or the live adapters.
No key is needed for CI or the local echo run. A live Jev run needs a TypeSafe key for JevHttpBackend, and the LLM baseline needs an OpenAI key, since the openai package is only imported during a live call.
Only synthetic tasks written inside the repository, in Portuguese with English keys. No customer data is included. The single Box task is a placeholder that the runner skips because it has no image asset or danger-zone header.
Three things on the same labeled tasks: whether the response obeys the typed schema (Choice, Score, Noul, Box), whether the decision agrees with the gabarito, and whether declared confidence tracks correctness.
Choice is scored by equality of the chosen key, Score by the nearest whole rubric level, and Noul by a 0.5 threshold where a probability of 0.5 or higher counts as yes. Invalid response types and transport errors leave the match null.
The client posts to /v1/systemone using either the jev-latest alias or a versioned id, currently jev-1.14.0. The client requires the usage field because the API marks it as mandatory.
It publishes no model results, covers no Box cases, and uses a small synthetic set of 45 tasks. It is a harness and methodology, not a production benchmark or a measurement of speed, cost, or hallucination rates.
More Jev resources like this
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Communityn9omi/jev-calibration-audit
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonIndependent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.