Are the benchmark numbers real or tuned on the test set?

Blend weight and temperature were chosen on validation, never on test, and the repo reports accuracy, calibration, latency, and coverage tables on the official test split plus the scripts to rerun them.

Why does a frozen probe beat the fine-tuned model here?

The frozen encoder with logistic regression scores above the fine-tuned run, matching an independent finding the docs cite; the shipped ensemble blends both, which is why the pipeline runs that baseline first.

Can I actually route on the confidence scores?

Yes for a slice: the top-confidence band covers about a quarter of decisions at high accuracy, which the docs present as an autoroute band with the rest escalated.

What breaks if I change encoders or rename labels?

Candidate labels ride in the input behind marker tokens with one shared head, so new questions and labels are inference-time data; a causal encoder option exists but needs different pooling than CLS.

Can I run it in CI without an API key?

Yes. The GitHub Action runs typed decisions on the runner's CPU with no API key and no per-call cost, and the state never leaves the runner. The first run downloads about 1.2 GB of ONNX files, checked against SHA256SUMS, then caches them.

What built-in workflows does the Action ship?

Four: security_incidents, agent_trace_observability, customer_service and invoice_processing. Answers from a built-in use the 0.697 ensemble; your own questions run the fine-tuned model alone at 0.6235, which the README warns can be confidently wrong on unfamiliar questions.

How is it trained and how long does that take?

It trains on a free Colab T4 in under 30 minutes with soft cross-entropy against annotator distributions plus a Brier term, with per-type temperature fitted on validation data. The PyTorch checkpoint is a 298 MB fp16 file split into six pieces on the release.

CommunityJev Demos & Experiments

Open Jev Typed Decision Engine

Open 150M encoder that reproduces the typed noul, choice, and score interface in one forward pass, trainable on a free GPU and runnable locally for nothing.

See highlights

Open github.com

Stars: 46
Language: Python
License: APACHE-2.0
Published by: Community
Author: intikhab49
Last verified: Oct 2026

intikhab49/open-jev-typed-decision-engine

User repository on GitHub

View on GitHub

Open reproduction of TypeSafe Jev: a 150M typed decision engine (noul/choice/score in one non-autoregressive pass, calibrated confidence). 0.697 vs Jev's 0.727, 2.5x better calibrated, 4x faster, free. Trains on a Colab T4 in 30 min.

Forks: 2
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

A 150M encoder that answers arbitrary typed questions about a state in one forward pass, with calibrated confidence. 0.03 behind TypeSafe Jev on its own benchmark, 2.5× better calibrated, 4× faster, $0. Results · GitHub Action · How it works · Quick start · What we learned · Limitations.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Ranking

Used inJev for Software & DevTools

Languages: Python

Highlights

Encodes state plus per-request questions with a single linear head read at label markers, so new schemas need no retraining or new weights.
Trains with soft cross-entropy against annotator distributions plus a Brier term, with per-type temperature fitted on validation data.
Reports near-hosted accuracy with lower calibration error and faster local per-case time on the public typed-decisions test split.
Ships a full pipeline from smoke test and baselines through training, calibration, evaluation, serving, and quantized export.

Watch out

Results are measured on the official typed-decisions test split with validation-chosen blend settings; the ensemble uses a fixed schema and confidence routing still needs threshold policy. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

OpenJev

Independent Jev-inspired decision API with choice, score, and noul primitives, shipping a local mock server plus Python and JavaScript SDKs for interface prototyping.

Jev Demos & Experiments#community#jevbest#catalog-import

Community

xingwudao/OpenJevPython

tinyjev

Local typed-decision models that answer Choice, Noul and Score in one forward pass on your own machine, with a Python API, local HTTP server and System One request shape.

Jev Demos & Experiments#community#jevbest#catalog-import

Community

ankit-aglawe/tinyjevPython

TypeLLM

941

Library that adds schema-guaranteed typed outputs to existing autoregressive LLMs without changing weights, served through SGLang or a hosted API.

Jev Demos & Experiments#community#jevbest#catalog-import

Community

TypeLLM/TypeLLMPython

open-medical-jev

Research recipe that fuses two frozen open readers into Jev-style yes/no judgments with a fit-free router, confidence gate, and candidate sets.

Jev Demos & Experiments#community#jevbest#catalog-import

Community

FeiLiuEM/open-medical-jevPython

jevify

Toolkit that makes a model behind an OpenAI-compatible endpoint or running in-process answer typed choice, score, and yes/no questions from next-token probabilities in Jev API shape.

Jev Demos & Experiments#community#jevbest#catalog-import

Community

fidecastro/jevifyPython

jev-forge

164

End-to-end toolkit that synthesizes decision data, trains and calibrates candidate scorers, evaluates fixed splits, and serves Jev-compatible inference for interactive web decisions.

Jev Demos & Experiments#community#jevbest#catalog-import

Community

zwliJay/jev-forgePython

From the Jev guides

Original write-ups on what this entry is for.

Per-Tick Decisions in Games

Sep 28, 2026

A game tick is a state plus a legal action list. How Clash Royale, Minecraft, and the earlier Mario, Pokémon, StarCraft, and drone loops ask Jev one step at a time.

Jev Demos & Experiments#gaming#agents#control

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai