Won't model-judged tests just be flaky?

Probabilities are calibrated, so 0.9 means 0.9, and the middle band refuses to decide: a 0.67 fails both toSatisfy and not.toSatisfy as ambiguous instead of silently passing. Sharpen the expectation or adjust thresholds.

How is this different from string matching or prose judges?

String matching checks substrings while prose judges return paragraphs to parse. Here each judgment is one Noul question over the output plus context, returning one number that a threshold turns into pass or fail.

How many API calls does a many-assertion suite make?

One per output, not one per assertion. Assertions on the same output in the same tick coalesce, and toSatisfyAll checks several expectations in a single request with failures listed first.

How do I run this in CI without a live key?

Record once locally to commit .jevtest/cache.json, then run CI in replay mode with no key and no network. A cache miss errors loudly, so changed outputs fail visibly until re-recorded.

How do I install and set it up for Vitest?

npm i -D jevtest, export TYPESAFE_API_KEY, add jevtest/vitest/setup to setupFiles with a 30000 ms test timeout, and import jevtest/vitest in a declaration file for TypeScript. Every matcher is async, so await the expect call.

How do semantic snapshots work?

toMatchSemanticSnapshot stores the first output in __semantic_snapshots__/ .jevsnap.json. Later an identical output passes with no call, a different one is classified cosmetic, behavioral or unclear, and only cosmetic at 0.8 or above passes; vitest -u or JEVTEST_UPDATE=1 rewrites the baseline.

Can I use it without a test framework?

Yes. The core exports satisfies, satisfiesAll and classifyDiff, which return the probability, model, latency and cached flag, plus getStats for requests, questions, cache hits, tokens and total latency.

CommunityJev Practices & Patterns

Jevtest — realZachi

Semantic test matchers for Vitest and Jest where plain-English expectations become single Noul judgments, with probabilities mapped to pass, fail, or ambiguous.

See highlights

Open github.com

Stars: 3
Language: TypeScript
License: MIT
Published by: Community
Author: realZachi
Last verified: Oct 2026

realZachi/jevtest

User repository on GitHub

View on GitHub

Semantic test matchers for Vitest and Jest, powered by TypeSafe's Jev model. Write expectations in plain English, get calibrated probabilities back.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

This is one of several Jev listings for the same project. The canonical entry is Jevtest — joshhu.

About this project

Semantic test matchers for Vitest and Jest. You write the expectation in plain English, jevtest asks TypeSafe's Jev model one narrow question about the output, and the matcher turns the returned probability into a pass or a fail. Jev does not generate text. It answers typed questions and returns a calibrated probability, usually in 150 ms to 1.5 s, which is cheap enough to run inside a test suite. The TypeSafe SDK is a dependency of jevtest, so there is nothing else to install.

Quoted from the project's README. Read the full README.

Tagscommunity awesomejev catalog-import

Works forJev for Classification

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

toSatisfy passes at 0.85 or above while not.toSatisfy passes at 0.15 or below, with the band between failing both as ambiguous.
Assertions on the same output in the same tick coalesce into one request; toSatisfyAll checks several expectations at once.
Stored-output baselines beside tests pass only cosmetic diffs at 0.8 or above, updated via -u or JEVTEST_UPDATE=1.
Content-hash caching plus record and replay modes let CI run offline, where a cache miss fails loudly.

Watch out

Needs TYPESAFE_API_KEY for live judgments, matchers are async and must be awaited, and TypeScript needs the setup import. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

Kev

Self-hosted Apache-2.0 decision engine with a Jev-style wire format that turns a model you point at it into typed choice, score, and noul answers.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

arjun988/KevTypeScript

jev-lm

Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

y0usaf/jev-lmTypeScript

claude-jev

Claude Code plugin that routes code review, debugging, planning, and search triage through typed Jev judgements with fixed thresholds applied in code.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

buchmark/claude-jevTypeScript

jev-agent-authorization

Starter kit putting Kinde identity plus Jev judgment in front of every agent tool call, with allow, hold-for-approval, or deny decided in code and written to a ledger first.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

kinde-starter-kits/jev-agent-authorizationTypeScript

jev-codes

CLI and agent commands that audit a git diff hunk-by-hunk against editable YAML standards packs, with Jev answering typed questions and the agent fixing flagged lines.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

Kushwho/jev-codesTypeScript

jevarena

Open arena pitting Jev against a chosen opponent on your question, hiding identities until you vote, then revealing latency, cost, and confidence data.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

chenmingtang830/jevarenaTypeScript

From the Jev guides

Original write-ups on what this entry is for.

Designing the State: What to Include and What to Drop

Oct 12, 2026

Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.

Jev Practices & Patterns#state#design#primitives

Read article

Noul, End to End: One Probability and a Dozen Thresholds

Oct 12, 2026

A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.

Jev Practices & Patterns#noul#primitives#confidence

Read article

Scoring Without Measuring: Bands, Rubrics, and Composite Triage

Oct 12, 2026

A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.

Jev Practices & Patterns#score#primitives#scoring

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai