User repository on GitHub
Using Jev as an evaluator.
- Forks
- 16
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Jev as a judge for agent evals
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
- Stars
- 108
- Language
- Python
- Reading time
- 4 min
- Published by
- Community
- Author
- danielgshea
- Last verified
- Sep 2026
About this project
Agent evaluators usually fall into two categories: deterministic code and LLM-as-a-judge. Code is fast and reliable but limited to behavior that can be expressed as explicit logic. LLM judges can evaluate open-ended agent behavior, but they add cost, latency, and variance. This project tests a third option: Jev, TypeSafe AI's System One decision model. We compare Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 on the same fixed agent runs, measuring binary accuracy, score reliability, cost, and latency.
Quoted from the project's README. Read the full README.
Used inJev for Software & DevTools
Highlights
- Across 500 repeated does_pass decisions per judge, Jev matched every human label for 100.0% accuracy versus 99.8% Terra, 96.4% Luna, and 80.0% Sonnet.
- Jev's mean per-case quality variance was 0.0000149; Luna was 433x higher, Terra 913x higher, and Claude 92x higher.
- Five cases (Seattle, Austin, Dublin, Tokyo, Springfield) were captured once and reused so only the judges introduced variation.
- The README calls this a small corpus with one human reviewer and states it is not a general ranking of judge accuracy.
Quickstart
bash
cp .env.example .env
# Add your API keys to .envWatch out
No license file, so reuse terms are unclear. Needs Python 3.13+, plus TypeSafe, Tavily, and workspace-scoped LangSmith API keys; the results come from five runs and one human reviewer.
Questions about this entry
Short answers drawn from the linked project or page.
Five examples. One weather agent with Tavily search, five cases in a LangSmith dataset, each run once and then frozen, with each judge evaluating those same five runs 100 times and a single human reviewer supplying the labels. The README says the result describes this experiment and is not a general ranking of judge accuracy, so read 100.0% versus 80.0% as a property of this corpus.
Both, with a caveat about how each was used. The experiment asks two things per run: a binary does_pass, which is what accuracy against the human oracle measures, and a continuous quality score from 0 to 1, used only to measure variance and never for accuracy. The authors also warn that low cost amplifies mistakes, so human review and judge alignment still matter.
The experiment says it cannot answer that. It calls the result observational rather than evidence that the architecture caused the lower variance, and offers one hypothesis: a model built to return bounded decisions may simply fit this task better than one built for autoregressive generation. It also notes that a repeatable evaluator can still be consistently wrong.
Both, but narrowly on price. Jev averaged $0.00035 per call at 0.44 s, $0.34 across the run, while Luna came to $0.00039 per call at 2.50 s. Terra was $0.00289 and Sonnet $0.02811 per call. The README notes these costs depend on the prompts, inputs, and provider pricing at the time of the run.
Three types exist: Noul, a probability that a yes/no judgment is true; Score, an ordered rubric score with probabilities and confidence; and Choice, one option with probabilities and confidence. This experiment asks two things per frozen run, a binary does_pass and a continuous quality score from 0 to 1. Human labels score only the binary decision, while the continuous score is used only to measure variance across the 100 repetitions.
Python 3.13 or newer, a Tavily key for the weather agent's web search, a TypeSafe key for the Jev evaluator, and a workspace-scoped LangSmith key with Gateway access. Copy .env.example to .env, fill in the keys, then uv sync. `uv run python main.py` runs the local evaluation without uploading anything; src/evals/offline_evals.py records a LangSmith experiment and src/evals/judge_reliability.py reproduces the repeated-judge benchmark.
The LangSmith experiment benchmark-jev-luna-terra-sonnet is recorded by id, with frozen cases, oracle labels and generated charts committed under assets/, and the LLM judges are named exactly: openai/gpt-5.6-luna, openai/gpt-5.6-terra and anthropic/claude-sonnet-4-6 through LangSmith Gateway, with Jev on langchain-typesafe==0.0.1a2. Limits: no temperature, top-p, seed or max tokens were set for the LLM judges, so provider defaults applied, and the metadata never exposed the hosted Jev service version.
More Jev resources like this
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonIndependent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Jev Practices & Patterns#community#python#benchmarks
Community
fstandhartinger/jevbenchPython
fstandhartinger/jevbenchPythonAn independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonIndependent measurement of whether ORDER BY over a Jev probability is defensible, pre-registering gates on pairwise inversion, Score ordinality, calibration, and negation and paraphrase invariants.
Jev Practices & Patterns#community#python#benchmarks
Community
yodablocks/jev-orderby-benchPython
yodablocks/jev-orderby-benchPythonA pre-registered, independently rescored audit of Jev's calibration and type safety: calibrated on CLINC150 (ECE 0.0204), systematically overconfident on Banking77 (ECE 0.0936), and zero type errors across 8,576 responses.
Jev Practices & Patterns#community#python#calibration
Community
jourdanlabs/assay-001Python
jourdanlabs/assay-001PythonAn independent baseline that reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass, reproducing Jev's interface pattern with open models rather than Jev's undisclosed model or training.
Jev Practices & Patterns#community#python#open-models
Community
TheoLeeCJ/SemIfPython
TheoLeeCJ/SemIfPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.