Zero false allows across 76 tries — is that a real zero?

No, and the README says so: zero in 76 is a bound of 3.9%, not a rate of zero. Held-out safe commands cleared only 38% of the time against 85% on the dev set, so the gate is conservative by design. Read the safety number as a ceiling and the clearance rate as the cost of that conservatism.

How is this different from just calling the Jev API?

It does not call it. The README states the project is independent, nothing here calls the Jev API, and nothing was trained on its output. What it borrows is the idea of a decision model answering typed questions with probabilities, and it runs that against any OpenAI-compatible endpoint returning logprobs, including a local Ollama model that needs no key.

Does the fine-tuned 0.6B judge actually beat the 8B?

On this machine's own commands, yes and by a lot: 656 of 733 safe cleared with 4 of 256 unsafe let through, against 308 and 1 for the prompted 8B, at 44 ms. Off that traffic it only ties the 8B, at 30 of 77, and lets 2 through. And a blind audit of 300 of its 535 clears found 6 wrong by the strict reading, so it missed the under-1% target it was held to.

Answer ordering changes the scores — does that make the gate untrustworthy?

It makes the self-check mandatory rather than the gate useless. Listing N before Y and changing nothing else moves every score, so the server answers canaries at startup and refuses to run if they come back wrong or if the judge is not the model the canaries were recorded against. --allow-unverified overrides that, on a threshold nobody measured for that judge.

How do I install it and run the offline tests?

It is not on npm, so install from GitHub with npm install github:liu-x27/XavierJev#v0.7.1; npm test runs 67 mocked checks with no model and no key, and the risk-gate dev eval runs offline against its allow-list.

What does the Claude Code plugin change about permissions?

It points the PermissionRequest hook at a local gate: a command scored safe is cleared without the prompt, but it never denies, every failure including a missing server is asked as usual, and it stays out of automatic modes.

What judge does the gate need, and how is it configured?

An OpenAI-compatible endpoint that returns logprobs; a local Ollama llama3.1:8b from the README works with no key, and llama.cpp's llama-server works if started with --no-jinja so its date template does not move the answers.

CommunityJev Agents & MCP

XavierJev

A TypeScript decision layer for agent loops that reads yes/no, one-of-n and rubric answers off one token's probabilities from a local logprobs model, with a permission gate shipped as a Claude Code plugin.

See highlights

Open github.com

Stars: 2
Language: TypeScript
License: MIT
Published by: Community
Author: liu-x27
Last verified: Oct 2026

liu-x27/XavierJev

User repository on GitHub

View on GitHub

A decision layer for agent control flow — yes/no, choice and rubric questions answered from one token's probabilities, measured against labelled sets — with a Claude Code permission hook and a trainable local judge

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

A decision layer for agent loops: yes/no, one-of-n and rubric questions read off one token's probabilities from a small local model, in tens of milliseconds, and measured against labelled sets before any of them is trusted. Project page · Three primitives · Four decisions · In Claude Code · On real traffic · Measurements.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification Jev for Routing Jev for Ranking Jev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

Nothing here calls the Jev API and nothing was trained on Jev output; the shape is borrowed from the decision-model design as an independent project.
The shipped gate cleared 1,181 of 4,000 real shell commands and every clear was read one by one: 6 should have been asked, about 1% at most.
Answer-order sensitivity is measured rather than assumed — choice() picks the listed-first option 48% of the time against 22% chance.
A regex beat the model on retry-a-failed-read, 36/36 against 29/36, so the regex ships, and a snake agent scores 43/133 on a raw board.

Watch out

This is not a Jev client. The README is explicit that the shape is borrowed from TypeSafe's System One design, that nothing here calls the Jev API, and that nothing was trained on its output. What it needs is an OpenAI-compatible endpoint that returns logprobs — a local Ollama qualifies and costs nothing — and it is not on npm, so installing from GitHub builds it. The figures are bounds, not rates: zero false allows across 76 held-out commands is a 3.9% ceiling, the 2,819 commands the gate held back were never read, and the fine-tuned 0.6B judge that cleared roughly twice what the prompted 8B clears only ties the 8B on commands written elsewhere and lets 2 through. A blind audit of 300 of its clears found 6 wrong by the strict reading, short of the under-1% target it was held to. Wording is fragile too: naming N before Y moves every score, which is why the server checks canaries at startup and refuses to run against an unverified judge unless you pass --allow-unverified. In Claude Code it never denies, only the command text reaches the judge and not the description the agent wrote, and no session of real use has been tested. llama-server needs --no-jinja because its default template shifts the answers. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

hermes-jev-plugin

Hermes Agent plugin exposing Jev as four coding-workflow tools for parallel judgments, binary checks, option routing and rubric scoring, with a bundled question-design skill.