User repository on GitHub
Pytest plugin for semantic assertions on LLM output using Jev's calibrated probabilities.
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Repos & SDKs
Pytest Jev
A pytest plugin for semantic assertions: jev.expect sends many holds and lacks claims about one text to Jev in a single request and fails visibly when a probability is uncertain.
- Stars
- 7
- Language
- Python
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- allebee
- Last verified
- Sep 2026
About this project
Test what your LLM app's output means, not the exact words. Semantic assertions for pytest, judged by TypeSafe's Jev, a model that returns calibrated probabilities instead of text. Same verdicts as Claude Sonnet 5 on the example tests, 5× faster and 110× cheaper. Benchmark · real runs of jev-1.13 through OpenRouter. Quick start · Usage · Benchmark · Examples. All the claims about one text go to Jev in a single request. When a prompt change breaks the reply, the failure shows which claims broke and how sure Jev was:
Quoted from the project's README. Read the full README.
Highlights
- All claims about one text go to Jev in one request; holds needs p >= 0.8 and lacks needs p <= 0.2, and unsure claims fail.
- Answers cache in .pytest_cache, so rerunning unchanged tests makes no requests and gives identical verdicts.
- On 12 example tests it matched Claude Sonnet 5's verdicts in 5.3 s versus 27.1 s, at $0.00017 versus $0.0192 per run.
- The jev fixture exposes holds, lacks, expect, choice, and score methods; each sends one request and returns an assertable result.
- Without a key, tests that use the jev fixture are skipped by default.
Quickstart
python
def test_refund_reply(jev):
reply = support_bot("I was charged twice for order #1042.")
jev.expect(
reply,
holds=["apologizes to the customer", "says the duplicate payment was refunded"],
lacks=["blames the customer", "asks for a password or a full card number"],
)Watch out
MIT-licensed and not affiliated with TypeSafe; needs Python 3.10+ and pytest 7.4+, plus a TYPESAFE_API_KEY or OPENROUTER_API_KEY, and asserted text is sent to the provider.
Questions about this entry
Short answers drawn from the linked project or page.
Twelve designed cases measuring speed, cost and agreement, not general accuracy, and the README says exactly that. On them Jev matched Claude Sonnet 5's verdicts in 5.3s versus 27.1s at $0.00017 versus $0.0192 per run. Measure a judge on your own labelled cases before trusting its thresholds.
Not on unchanged tests. Answers are cached by model, text, context and question in .pytest_cache, so a rerun makes no requests and gives identical verdicts; pass --jev-no-cache to ask again. Note the default model is jev-latest, which moves when TypeSafe ships, so pin jev-1.13 when runs must be reproducible.
Tests using the jev fixture get the jev marker automatically and are skipped with a reason, which can quietly green a build that stopped testing anything. Pass --jev-require, or set jev_require = true, so a missing secret fails instead. pytest -m "not jev" runs everything else offline.
Yes, through context. Extra state such as a policy or retrieved documents goes in and a claim names it in backticks, for example lacks(reply, contradicts the policy in `policy`, context with the policy text). The text under test is always the text argument, so context cannot use that key.
Yes. The threshold defaults to 0.8, where holds needs p at least 0.8 and lacks needs p at most 0.2, and it must sit between 0.5 and 1. Set it per call, per test with the jev marker, or for the whole run.
Alongside holds and lacks for one claim, expect bundles many claims into a single request, choice picks between labelled options, and score rates text on ordered levels. Comparing a choice against a label that is not an option raises an error.
Yes. Requests go through the session-scoped jev_client fixture, so you can override it in conftest with anything exposing system_one(state=, questions=, model=), such as a fake for offline tests.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Agent Handoff Gate: agents verify what they hand off
X: Jev as an LLM guardrail inside Agentgateway
X: Vercel's fx safety reviewer, measured at 18× faster
Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.
More Jev resources like this
OpenLayer's eval and guardrail library that replaces LLM judges with Jev-style decisions, packing every check for an agent trace into one request.
Jev Repos & SDKs#community#python#evaluation
Community
openlayer-ai/jevalsPython
openlayer-ai/jevalsPythonA Python library that turns Jev-style decision models into ordinary function calls: check returns a bool, classify picks one option, label picks every option that applies, and score rates on a scale, with probabilities behind every answer.
Jev Repos & SDKs#community#github#python
Community
jlowin/vibecheckPython
jlowin/vibecheckPythonA Python library that adds .jev accessors to pandas and Polars: ask a natural-language question per row and get labels, scores, and full probability distributions back.
Jev Repos & SDKs#community#python#data
Community
ktaletsk/jevframePython
ktaletsk/jevframePythonA Python library that reranks search results and filters retrieved documents with Jev, scoring each candidate's usefulness as evidence and dropping those below a configurable threshold before RAG context is assembled.
Jev Repos & SDKs#community#python#reranking
Community
hotchpotch/jev-rerankerPython
hotchpotch/jev-rerankerPythonA frozen decision server that serves Gemma 4 31B — quantized to FP8 at startup — in TypeSafe's /v1/systemone format, reading the next-token distribution instead of generating tokens. It needs one FP8 NVIDIA GPU with 80 GB and has no authentication.
Jev Repos & SDKs#community#github#python
Community
krishna-gogineni-765/deck31bPython
krishna-gogineni-765/deck31bPythonAn experimental TypeScript harness that connects evidence to Jev judgments, permitted actions, and verified outcomes, with pluggable data sources and tools, local JSONL traces, and zero runtime dependencies.
Jev Repos & SDKs#community#typescript#agents
Community
Kevthetech143/super-jevPython
Kevthetech143/super-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.