Is this a real replacement for Jev or just a matching shape?

Shape-compatible, not capability-identical. The same {state, model, questions} shape goes in and {model, answers, usage} comes out, but the default scorer is deterministic and carries no intelligence until you switch on the frozen-model scorer.

Which model size should I start with locally?

Start light with Qwen2.5-3B-Instruct around 6GB on a 16GB Mac. Move to Qwen3.5-9B around 18GB on a GPU box when you need the higher reported accuracy and can pay 0.40s per question.

How do I point my existing SDK code at it?

Keep your code and set base_url to http://127.0.0.1:8000. The drop-in proof shows the official typesafe-sdk working with only that change, including a lockout case over an SSH tunnel.

Why do choice and score lag noul on small models?

That matches the published evals: noul stays near the top across sizes while choice and score collapse below roughly 1B params, which is why no sub-1GB tier is shipped and the floor stays at 1.5B.

How do I start it in one line?

The install script clones the repo to ~/jev-local, starts the API on http://127.0.0.1:8000 via Docker Compose, waits for health and runs a smoke test. Without Docker, run `pip install .` then `jev-local-serve --port 8000`. The endpoint serves POST /v1/systemone with typed noul, choice and score questions.

How do I switch from the default scorer to a real model?

The default scorer is deterministic and carries no intelligence. Set JEVLOCAL_SCORER=hf with the .[hf] extra and JEVLOCAL_MODEL, for example JEVLOCAL_MODEL=Qwen/Qwen3.5-9B, to use the frozen-model logprob scorer. JEVLOCAL_CHAT=1 wraps the prompt in the model's native chat template, which lifts set1 but stays opt-in because it is mixed on the harder set3.

What shape and limits does the local API enforce?

The same shape as the TypeSafe endpoint: {state, model, questions} in and {model, answers, usage} out, with choice capped at 1 to 255 options, score at 2 to 10 levels, and noul criteria keys limited to true and false. Usage tokens are estimates. For the browser demo use 127.0.0.1 rather than localhost, since CORS is only enabled for that.

CommunityJev Demos & Experiments

Jev Local — us

Local server emulating the Jev System One endpoint at POST /v1/systemone with typed noul, choice, and score questions, so the official SDK works with only base_url changed.

See highlights

Open github.com

Stars: 8
Language: Python
Published by: Community
Author: us
Last verified: Oct 2026

us/jev-local

User repository on GitHub

View on GitHub

Local Jev-compatible evaluation server: POST /v1/systemone with typed noul/choice/score, open weights, no waitlist

Forks: 2
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

This is one of several Jev listings for the same project. The canonical entry is Jev Local — Argos1111.

About this project

Local Jev-compatible evaluation server: POST /v1/systemone with typed noul / choice / score questions, probabilities, and confidence. No waitlist, no API key, no closed weights. This clones the repo to /jev-local, starts the API on ` via Docker Compose, waits for health and runs a smoke test. Afterwards: Or point the official SDK at it with baseurl=" No Docker? pip install . then jev-local-serve --port 8000`.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification Jev for Ranking Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

One-line install clones the repo, starts Docker Compose on 127.0.0.1:8000, waits for health, and runs a smoke test.
Default scorer is deterministic and carries no intelligence; JEVLOCAL_SCORER=hf switches to frozen-model logprobs.
Full Qwen3.5-9B setup needs around 18GB of weights while the light Qwen2.5-3B option fits a 16GB Mac.
Enforces the same shape and limits as the TypeSafe endpoint, including choice, score, and noul criteria rules.

Watch out

Small-model accuracy and shuffle sensitivity are documented in-repo, and usage tokens are estimates, so treat the eval tables as repo-reported. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

open-jev-typed-decision-engine

Open 150M encoder that reproduces the typed noul, choice, and score interface in one forward pass, trainable on a free GPU and runnable locally for nothing.