A Python toolkit from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models; it returns typed scores, thresholds and status instead of text judgments.

Jev is the default judge backend: the toolkit assembles the evidence, has the decision model judge it, and returns typed results your code can branch on. The quickstart needs a TypeSafe API key.

Can I use a different judge backend?

Yes: Microsoft Decision-1 on OpenRouter through OPENROUTER_API_KEY, or the native OpenAI Decisions API through OPENAI_API_KEY, both through the same backend as Jev.

What are the tool guards for?

They check proposed tool calls before they run, so an agent's actions can be gated by typed decisions rather than by a text model's critique.

How do I install and start?

With Python 3.11+, run python -m pip install typed-evals, export TYPESAFE_API_KEY='your-key', then evaluate a response against the evidence used to generate it.

Does it support human-label calibration?

Yes; calibration against human labels is documented as an optional part of the framework, alongside evaluation of recorded agent executions and the metric presets.

What did the digest not capture?

The captured README cuts off in the first Python example, so full request shapes, pricing and rate limits were not available; check the repository guides and examples before production use.

Jev directory

CommunityJev Benchmarks & Evaluations

Typed Evals

A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.

See highlights Open github.com

Stars: 16
Language: Python
License: MIT
Reading time: 2 min
Published by: Community
Author: TrustifAI
Last verified: Oct 2026

trustifai/typed_evals

Organization repository on GitHub

View on GitHub

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

Forks: 1
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity github python evaluation calibration

Works forJev for Evaluation Jev for Verification Jev for Decision gating

Used inJev for Software & DevTools

Languages: Python
Platforms: API OpenRouter
Capabilities: Open sourceRequires API key

Highlights

Evaluate responses. Guard actions: it judges generated responses and recorded agent executions, and checks proposed tool calls before they run.
Jev is the default backend; Microsoft Decision-1 on OpenRouter and the native OpenAI Decisions API work through the same backend.
Takes evidence through a metric panel and returns typed scores, thresholds and status rather than free-text critiques.
Start from presets, then bring your own metrics, thresholds or judge backend as the application grows; calibration against human labels is optional.
Installs with pip on Python 3.11+, sets TYPESAFE_API_KEY (or OPENAI_API_KEY), and is MIT licensed with CI and a PyPI package.

Quickstart

bash

python -m pip install typed-evals
export TYPESAFE_API_KEY='your-key'

Watch out

README reviewed 2026-10-17; the digest truncates at the start of the first Python example, so the concrete request shape, guard configuration and calibration workflow could not be quoted here. Metadata shows 16 stars, MIT, created 2026-09-20 and pushed 2026-10-15, with tags for LangChain, CrewAI and Microsoft Agent Framework integrations.

Questions about this entry

Short answers drawn from the linked project or page.

From the Jev guides

Original write-ups on what this entry is for.

The trust ledger: measuring confidence your domain has earned

Oct 16, 2026

ASSAY-001 calibrated Jev on CLINC150 but found it overconfident on Banking77 — one audit, two verdicts. How calibration research measures the model-and-domain pair, discounts stated confidence, and signs the evidence.

Jev Benchmarks & Evaluations#calibration#evaluation#open-models

Read article

What the First Month of Jev Papers Found

Oct 14, 2026

More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.

Jev Practices & Patterns#research#evaluation#calibration

Read article

Fine-Tuning and Calibration: Making an Open Checkpoint Yours

Oct 9, 2026

Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.

Jev Practices & Patterns#fine-tuning#calibration#open-models

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Typed Evals

Highlights

Quickstart

Questions about this entry

What is Typed Evals?

How does it use Jev?

Can I use a different judge backend?

What are the tool guards for?

How do I install and start?

Does it support human-label calibration?

What did the digest not capture?

Related terms

More Jev resources like this

From the Jev guides

From the Jev AI model community