Question 1

How can frozen models rival a hosted decision model with no training?

Accepted Answer

Nothing is trained: two off-the-shelf quantized readers vote per option through fixed prompts, and fixed-weight fusion plus a two-constant router turns agreement into confidence, release, and set outputs.

Question 2

Which compute mode should I run when latency matters?

Accepted Answer

Use the single-readout fast preset for lowest latency, the general cascade for large compute savings at high released precision, and the strict preset when you need higher gate precision.

Question 3

Can I swap in a newer base model without rebuilding everything?

Accepted Answer

Yes by design: swap the reader, refit the two small routing constants, and rerun the verification pass on your data while keeping the protocol, router, and guarantees unchanged.

Question 4

Why does calibration still need my own data before deployment?

Accepted Answer

Agreement discounts, temperature, gates, and conformal sets are fit on one distribution; the docs require recalibration on yours and warn that shifted data breaks the guarantees.

Question 5

What are the numbers behind the Jev-level claim?

Accepted Answer

On the 600-item China NMLE 2021 paper it scores 0.8883 against hosted Jev 1.13.0 at 0.8967 and OpenJev at 0.8800, with the US paper at 0.8634 against Jev's 0.8833 and the India paper at 0.8100 against Jev's 0.8133. On dev-300 the two-model mean fusion is 0.8167 with unanimous router coverage at 78.7 percent, and on JevBench public 82.9 percent coverage at 0.9310. Laya's released checkpoints score at or near chance on the same papers.

Question 6

How does it read a judgment out of a frozen model?

Accepted Answer

Two readout structures. The pair readout is primary: ask the reader whether the candidate answer is correct, reply yes or no, then read the yes/no token probabilities so the per-option signal is Δ = max logprob(yes) minus max logprob(no). The choice readout is secondary: all options in one prompt, reading letter probability mass. Both prompts, decoding parameters and tokenizer notes are pinned in docs/protocol.md, which is the contract, and src/open_medical_jev/protocol.py has to stay in sync with it.

Question 7

What does the toolchain look like, and how heavy is it?

Accepted Answer

scripts/setup_env.sh creates a venv and editable install, scripts/download_models.sh q4 fetches GGUF weights of roughly 17 GB plus 22 GB, and scripts/serve_model.sh 27b and 35b start llama.cpp on 127.0.0.1:10361 and 10362 on a single 24 GB GPU, with the MoE experts partly on CPU. The core pipeline has no third-party dependencies and selftest runs 16 dependency-free cases. CLI commands cover selftest, demo, check-server, verify-tokenizer, read, evaluate, calibrate and flip.

Open Medical Jev

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community