Article
CommunityJev Benchmarks & Evaluations
JevEval: Jev-as-a-Judge
DeepEval's custom evaluation metric that has Jev answer bounded questions about a test case, then maps its calibrated probabilities into a deterministic weighted score, with no generated verdicts or JSON recovery.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Author
- Jeffrey Ip
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- You give it the test-case fields Jev may see, bounded questions, and optional weights; Jev answers all of them in one request.
- Three question types: Noul, Score, and Choice, each matching one of Jev's decision primitives and mapping to a value from 0 to 1.
- The score is a fixed weighted mean over probabilities; include_reason defaults to a reason assembled from the outcomes, not generated text.
- With strict_mode=True the metric returns 1 only when every applicable question passes in its best possible way, otherwise 0.
- score_breakdown stores each question's type, weight, probabilities, mapped 0-1 value, and whether it applied.
Watch out
The blog post does not print a licence; the Jev News desk describes JevEval as Apache-2.0 and notes it did not install DeepEval. Jev still needs a TypeSafe key, and the docs' 0.461 walkthrough value is illustrative. A desk test scored the invented-breeze case at 0.254 against 0.901 for the clean answer.
Questions about this entry
Short answers drawn from the linked project or page.
A custom DeepEval metric for LLMTestCase that separates evaluation into three jobs: you define the logic, Jev makes the decisions, and DeepEval computes a fixed weighted mean. It produces no generated verdicts and needs no JSON recovery from an LLM.
Give JevEval the test-case fields Jev may see, a list of bounded questions, and optional weights per question. Jev receives the selected fields as structured state and answers every question in one request.
Noul asks whether a proposition is true and returns P(true). Score places the response on ordered, descriptive levels and normalizes the expected position to 0-1. Choice selects among unordered behaviours with credit from 0 to 1 each.
A Choice option can be mapped to None, such as nothing_missing. If Jev places at least half of the probability on not-applicable options, that question drops out of the score instead of rewarding or penalizing the response.
With strict_mode=True the metric returns 1 only when every applicable question passes in its best possible way, otherwise 0, with the threshold set to 1. The page recommends it for hard constraints such as every citation being supported.
No. By default JevEval attaches a reason assembled from the outcomes themselves, one line per question with its result, probability, and confidence. The same answers always produce the same reason, and there is no LLM in the metric.
Four questions about a weather answer: whether every fact came from a tool call, whether returned values were reported accurately, how grounded the response was overall, and how it handled information the tools did not return, with the first double-weighted.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Microsoft-Decision-1 against Jev on a private eval
X: Vals AI's independent evaluation of Jev
X: An eval suite for System One models
X: Jev and Laya play chess instead of benchmarks
More Jev resources like this
One Python interface for evaluating what LLM apps and agents say and do. Its CascadeEvaluator asks a decision model first — Jev, OpenAI's Decisions API, or any model served at /v1/systemone — keeps confident answers, and sends only the rest to an LLM judge.
Jev Benchmarks & Evaluations#community#github#python
Community
grounded-ai/grounded_aiPython
grounded-ai/grounded_aiPythonA CLI from Pinecone's DevRel team for testing Agent Skills and docs against tasks: run tasks across Claude, Copilot and Gemini in local or Modal sandboxes, compare with-skill, without-skill and with-docs variants, and grade with Claude or a TypeSafe System One backend.
Jev Benchmarks & Evaluations#community#github#python
Community
pinecone-io/cultivarPython
pinecone-io/cultivarPythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonA Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.