Jev Rs — yijunyu

It is the harness around models: a logprob scorer over any decoder, adapters for trained specialists, a TypeSafe wire server, and one eval and calibrate stack scoring them side by side. Single-technique models plug in as backends rather than competing harnesses.

Start llama-server with a GGUF, point jev at it, and ask with state plus choice, score and noul flags sharing one cached prefix ending in Answer:. Nothing is decoded and output token usage stays zero, with templates for non-ChatML models.

Run jev mcp over stdio with the judge tool from Claude Code, Codex, Grok Build or OpenCode configs, and instruct the agent to use it for routing, triage and yes-no checks with a confidence cutoff around 0.8. TypeSafe SDKs can target jev serve via base URL.

Eval reports accuracy, Brier, top-label ECE, coverage at 5% error, latency and token cost, and calibrate fits per-bucket temperatures into calibration.json. The digest tables show small models saturating classification while larger ones improve yes-no and ordinal quality.

The state is rendered once into a shared prefix and each question becomes a suffix ending in an answer cue with lettered options, so the backend returns raw next-token log-probabilities for those labels. Nothing is decoded and output token usage stays zero. A restricted softmax over the labels, divided by a fitted temperature per question type and option count, becomes the final probability.

A permutation flag: you ask it to average several rotations of the option order, which spreads out position effects instead of trusting a single ordering. The README treats this as the mitigation specifically because the backend returns probabilities that depend on where each option sits in the prompt.

On 75 hand-labelled decisions, a 4B model was fastest at 75 ms median and decent on choice but poorly calibrated, with fitted temperatures of three to six; a larger hybrid model reached the highest accuracy and near-calibrated confidence at 700 ms; and a huge streamed model gave the best yes/no answers but the worst ordinal ones. Larger models gated about twice as many decisions at a five percent error budget.