Are the probabilities it prints calibrated enough to threshold on?

Not across the board, and the README is upfront about which is which. A boolean answer is a plain P(true) and is marked uncalibrated. Confidence does exist for choice and score, but you only see it with --full, at jev.answers. .confidence when calling TypeSafe directly and typesafe.confidence through the gateway. Either way, calibrate against your own labeled examples before a number drives an automated branch.

Can I use this in CI without a config file at all?

Yes. Precedence is CLI flag, then environment, then the config file, so a job can export the key and skip the file entirely. The exit codes make failures distinguishable: 1 for provider or network failure, 2 for bad usage or invalid questions, 3 for missing or rejected credentials, with 0 on success. Errors print as JSON on stderr so stdout stays clean for results, and jev-cli doctor --offline checks configuration without a network call.

Should I stay on jev-latest or pin a version?

Pin it once your thresholds are calibrated. The model setting takes jev-latest, jev-preview or a pinned release such as jev-1.13.0, and the README notes the aliases move when a release ships, so a threshold tuned against one build can drift under the next. Pin the release and re-check your thresholds when you deliberately upgrade.

Why route through Vercel instead of calling TypeSafe directly?

Both are first-class. The default provider jev calls TypeSafe AI directly with model jev-latest; setting provider vercel plus a gateway key switches to the Vercel AI Gateway with model typesafe-ai/jev, which is convenient if you already front traffic through the gateway. One trap to know: JEV_CLI_API_KEY applies to whichever provider is active, so if both are configured the same key can be sent to both unless you use the provider-specific environment variables.

How do I install jev-cli and which runtime does it need?

Install with bun install -g @jtsang/jev-cli or npm install -g @jtsang/jev-cli. The published binary runs on Node 22+ or Bun.

How do I point jev-cli at my API key?

Run jev-cli config init, then jev-cli config set providers.jev.apiKey , then jev-cli doctor to confirm the key and model work. Pass --offline to check the configuration without a network call.

What do the jev-cli exit codes tell me?

0 is success, 1 is a provider or network failure, 2 is bad usage or invalid questions, and 3 is missing or rejected credentials. Errors print to stderr as JSON, leaving stdout for results.

CommunityJev Tools & CLIs

Jev CLI — jtsang4

A CLI that sends state plus typed questions to Jev and prints structured JSON, with boolean, choice and score question types, a documented exit-code contract for CI, and an optional Vercel AI Gateway transport.

See highlights

Open github.com

Stars: 3
Language: TypeScript
License: MIT
Published by: Community
Author: jtsang4
Last verified: Oct 2026

jtsang4/jev-cli

User repository on GitHub

View on GitHub

CLI for TypeSafe AI's Jev evaluation model — typed questions in, structured JSON answers out

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

This is one of several Jev listings for the same project. The canonical entry is Jev CLI — shaharia-lab.

About this project

A command-line interface for Jev, TypeSafe AI's evaluation model. Give it a state and some typed questions; get structured JSON back. Jev is a "System One" model: instead of generating prose, it judges shared state against typed questions and returns choices, scores, and probabilities your code can branch on. That makes it a good fit for classification, routing, rubric scoring, and automated verification — and a poor fit for writing text, which it cannot do. The published binary runs on Node 22+ or Bun.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

Three question types from JSON: boolean returns P(true), choice the label with per-option probabilities, score a weighted position.
Both inputs accept files or stdin, so piping a diff into a question file is the natural shape for a check in CI.
Exit codes are separated for scripts: 1 provider or network failure, 2 bad usage or invalid questions, 3 missing or rejected credentials.
Config lives in ~/.jev-cli/config.yaml at 0600, precedence runs CLI flag over environment over file, and keys are masked unless --show-secrets.

Watch out

Boolean probabilities are explicitly labelled uncalibrated, so a number near 0.98 on a yes/no question is not a 98 percent guarantee. Confidence exists for choice and score answers but only surfaces under --full, at jev.answers.<id>.confidence on the direct provider and typesafe.confidence through the gateway. A score is a fractional position in [0, levels-1], the probability-weighted mean rather than an index, which matters if you treat it as a label. Configuration trap: JEV_CLI_API_KEY applies to whichever provider is active, so with both jev and vercel configured the same key can be sent to both unless you set the provider-specific variables. The model field takes jev-latest, jev-preview or a pinned release such as jev-1.13.0, and the README advises pinning once your thresholds are calibrated because the aliases move when a release ships. The published binary runs on Node 22+ or Bun, questions are evaluated in parallel and in isolation so each must stand alone, and the package ships an agent skill. MIT. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-recall

Python memory filter that judges every stored memory against the current request with one calibrated yes/no per memory in a single Jev request, keeping everything above a threshold.