CommunityJev Benchmarks & Evaluations
Jev-as-a-Judge for Agent Evals
LangChain's LangSmith team benchmarks Jev as an agent evaluator against GPT-5.6 Luna, Terra, and Claude Sonnet 4.6 on five captured weather-agent runs. Jev matched the human oracle on all 500 binary pass/fail decisions with 92-913x lower quality-score variance and cost $0.00035 per call.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Jev matched the human oracle on all 500 repeated binary does_pass decisions; Terra 99.8%, Luna 96.4%, Claude Sonnet 4.6 80.0%.
- Mean per-case quality-score variance of 0.0000149, described as 92-913x lower than the three LLM judges.
- Averaged 0.44s latency and $0.00035 per call; $0.34 total versus $28.17 for Claude in the experiment.
- Ran on a fixed LangSmith dataset of five captured weather-agent runs from a Deep Agents harness.
Watch out
A narrow test by LangChain on five fixed weather-agent runs, using langchain-typesafe 0.0.1a2 with the Jev service version unrecorded. The post says results still need to be shown on other agents and production workflows.
Questions about this entry
Short answers drawn from the linked project or page.
Jev matched the human reviewer's oracle on all 500 repeated binary does_pass decisions. Over the same five captured runs, GPT-5.6 Terra matched 99.8%, GPT-5.6 Luna 96.4%, and Claude Sonnet 4.6 80.0% of decisions.
Jev averaged $0.00035 per call with 0.44s average latency, $0.34 in total for the experiment versus $28.17 for Claude Sonnet 4.6. The authors note low cost can amplify mistakes, so human review still matters.
Jev's mean per-case quality-score variance was 0.0000149, which the post describes as 92-913x lower than GPT-5.6 Luna, Terra and Claude Sonnet 4.6. The binary does_pass decisions are judged separately from those quality scores.
A System One model is built to make fast, structured decisions software can use directly: it evaluates a state and returns typed answers and probabilities. Jev does not generate text, unlike the autoregressive LLM judges it was tested against.
A fixed LangSmith dataset of five captured weather-agent runs from a Deep Agents harness. The authors call it a narrow test by LangChain and say the results still need to be shown on other agents and production workflows.
The experiment used langchain-typesafe 0.0.1a2, and the Jev service version is not recorded in the post.
They describe it as an early, narrow test on five fixed runs and say results still need to be shown on other agents and production workflows. Low cost can also amplify mistakes, so the authors stress that human review still matters.
More Jev resources like this
24.1kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPython3GitHub stars
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPython1GitHub stars
Python CLI that runs declarative YAML, JSON or JSONL test suites natively on Jev Noul, Choice and Score decisions for RAG faithfulness, hallucination detection and agent safety, with watch mode, parallel batching, a pytest plugin and GitHub Actions PR reporting.
Jev Benchmarks & Evaluations#community#cli#ci-cd
Community
00200200/jev-judgePython
00200200/jev-judgePythonA bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev community
Posts from builders shipping with Jev right now.