Skip to content
JevDirectory.org
CommunityRepos & SDKs

jevals

OpenLayer's eval and guardrail library that replaces LLM judges with Jev-style decisions, packing every check for an agent trace into one request.

Category
Repos & SDKs
Format
—
Published by
Community
Author
OpenLayer
Added
2026-09-25
Last verified
2026-09-25

Highlights

  • The quickstart weather trace runs nine checks in one request: 1,423 tokens, $0.00006, and 0.50 s.
  • Ships 37 evals across agent, security, and quality groups, plus YAML gates with allow, escalate, and block policy.
  • Against Ragas on the same sample it reports 1 request and $0.03 per 1k, versus 6.0 LLM calls and $2.60.
  • Backends resolve from the environment: TypeSafe direct, Vercel AI Gateway, Kev, Laya, or OpenRouter emulation.

Quickstart

python
from jevals import evaluate
from jevals.agent import Grounded, StayedInScope
from jevals.quality import AnswerRelevancy, Completeness

report = evaluate(
    {"messages": messages, "tools": tools},
    [Grounded(), StayedInScope(), AnswerRelevancy(), Completeness()],
)

Watch out

Alpha software: the TypeSafe direct backend has only been tested against a mock and the adapters against fakes, and probabilities differ per backend, so calibrate thresholds yourself.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: Agent Handoff Gate: agents verify what they hand off

X: Jev as an LLM guardrail inside Agentgateway

X: Vercel's fx safety reviewer, measured at 18× faster

Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.

Related terms

Glossary definitions related to this entry.

More like this

3GitHub stars
A pytest plugin for semantic assertions: jev.expect sends many holds and lacks claims about one text to Jev in a single request and fails visibly when a probability is uncertain.
Repos & SDKs#community#python#evaluation
70GitHub stars
A Python toolkit that puts Jev, Laya, or Cua-S1 at the center of browser, computer-use, robotics, and game agents, with a CLI and an MCP server.
Repos & SDKs#community#python#agents
A DSPy fork whose @typesafeify decorator routes decision-shaped signature fields to Jev, so bool, Literal, and scored outputs come back typed while freeform fields still use the LM.
Repos & SDKs#community#python#sdk
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

A really smart switch statement

hype-free explanation of jev: jev does not replace gpt / claude jev is just a *really* smart switch statement like if 2016 ml classifiers got 2026 levels of intelligence it's a new* type of tool that will make a lot of workloads insanely fast, cheap, and accurate * = and by Show more

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

When a designer gets Jev

Full Jev video tutorial

The case against Jev-scored compaction

This is a terrible compaction strategy that fundamentally doesn't understand how compaction and context management work. Seems like a lot of people are confused so let's break this down. 1. Compaction isn't a filter The role of compaction is to clean up history to keep the Show more

tamara
tamara
@tamarajtran

found the perfect use case for @typesafeai Jev: instant compaction in 2026, why is compaction still a summarization prompt? Jev can make it instant by scoring every tool call and dropping what’s irrelevant

Reply

Classifying rows in DuckDB

A playable 16-judgment demo