Organization repository on GitHub
Evals that meet you where you are. For AI that's grounded.
- Forks
- 7
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Benchmarks & Evaluations
Grounded AI
One Python interface for evaluating what LLM apps and agents say and do. Its CascadeEvaluator asks a decision model first — Jev, OpenAI's Decisions API, or any model served at /v1/systemone — keeps confident answers, and sends only the rest to an LLM judge.
- Stars
- 71
- Language
- Python
- License
- MIT
- Reading time
- 2 min
- Published by
- Community
- Author
- grounded-ai
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- CascadeEvaluator asks a decision model first — Jev, OpenAI's Decisions API, or any model served at /v1/systemone — and only uncertain answers reach the judge.
- On 200 agent turns: threshold 0.7 sent 37% to the judge at 1.28 s and $2,997 per 1M checks, versus 2.59 s and $7,751 for Sonnet 4.6 alone.
- The README reports 51% faster and 61% cheaper at threshold 0.7 with accuracy 0.695, against 0.675 for the judge alone.
- Install with pip install grounded-ai for LLM judges (OpenAI, Anthropic, Bedrock) or pip install "grounded-ai[jev]" to add decision models.
- MIT, 71 stars, 7 forks, Python, created 2024-06-14, last pushed 2026-10-16; a Colab notebook runs a full cascade.
Quickstart
python
from grounded_ai import CascadeEvaluator
from grounded_ai.backends.jev import HALLUCINATION, JevInput
cascade = CascadeEvaluator(
decision_model="jev/jev-latest",
judge="anthropic/claude-haiku-4-5",
min_confidence=0.7,
)Watch out
README reviewed 2026-10-17 and truncated mid-command in the Install block, so the decision-model extras and the full setup are not in the digest. The cascade timings, costs, and accuracies are the project's own benchmark on 200 agent turns against Claude Sonnet 4.6; no independent replication appears in the excerpt.
Questions about this entry
Short answers drawn from the linked project or page.
One Python interface for evaluating what LLM apps and agents say and do: use a frontier model as the judge, a small local model, or a decision model, and swap between them without rewriting your evals.
Its CascadeEvaluator asks a fast decision model first — the README names Jev at docs.typesafe.ai, OpenAI's Decisions API, or any model served at /v1/systemone — keeps confident answers, and sends only the uncertain ones to the LLM judge.
On 200 real agent turns asking whether the agent should have called a tool or replied: at threshold 0.7, 37% went to the judge and cost $2,997 per 1M checks instead of $7,751, at 1.28 s instead of 2.59 s, with accuracy 0.695 versus 0.675.
pip install grounded-ai installs the LLM judge path for OpenAI, Anthropic, and Bedrock; pip install "grounded-ai[jev]" adds decision models. The digest truncates the second command mid-line, so the extras list may include more.
The example pairs decision_model="jev/jev-latest" with judge="anthropic/claude-haiku-4-5"; the README also shows "openai-decisions/gpt-6-luna" and "systemone/agent-decision-9b" with a base_url, and the notebook serves agent-decision-3b.
Yes: the table covers 200 real agent turns judged against Claude Sonnet 4.6 alone, and the digest shows no independent replication. Treat the 51% faster and 61% cheaper claims as the project's own benchmark.
The grounded-ai organization's MIT-licensed Python repo: 71 stars, 7 forks, created 2024-06-14, and last pushed 2026-10-16. The README links a blog write-up and a benchmark script for the tool-call cascade.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Microsoft-Decision-1 against Jev on a private eval
X: Vals AI's independent evaluation of Jev
X: An eval suite for System One models
X: Jev and Laya play chess instead of benchmarks
More Jev resources like this
A CLI from Pinecone's DevRel team for testing Agent Skills and docs against tasks: run tasks across Claude, Copilot and Gemini in local or Modal sandboxes, compare with-skill, without-skill and with-docs variants, and grade with Claude or a TypeSafe System One backend.
Jev Benchmarks & Evaluations#community#github#python
Community
pinecone-io/cultivarPython
pinecone-io/cultivarPythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonDeepEval's custom evaluation metric that has Jev answer bounded questions about a test case, then maps its calibrated probabilities into a deterministic weighted score, with no generated verdicts or JSON recovery.
Jev Benchmarks & EvaluationsArticle#community#evals#deepeval
Community
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonGTM workflow experiments with Jev from deepdave98: weekly datasets, prompts, and benchmark results that test Jev against Claude on lead triage and routing, deal risk, reply triage, MEDDPICC from call summaries, duplicate accounts, and closed-lost reactivation.
Jev Benchmarks & Evaluations#community#github#gtm
Community
deepdave98/jev-playgroundPython
deepdave98/jev-playgroundPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.