Question 1

Is this a client for the Jev API, or a completely separate model?

Accepted Answer

A separate model. There is no hosted API call in the pipeline: it loads Qwen3-1.7B as a frozen backbone, caches hidden states for every candidate path, trains a small decision head on those vectors and serves the result locally with minojev serve. What it reproduces is the decision interface, not the service: choice with 2 to 255 candidates, boolean, and score with 2 to 10 ordered levels, each returning a complete probability distribution and decode_steps: 0.

Question 2

Beating generation 95.8% to 80.0% sounds rigged. How is that comparison set up?

Accepted Answer

It is the README's own benchmark, not an independent one, and the setup favours the decision path in ways you should price in. Same backbone, same questions, same prompts, and the generative baseline runs through the chat template with thinking disabled and is told to answer with a single letter. The suite is 120 balanced decisions drawn from banking77, CLINC150, Amazon Polarity and GSM8K verification. ECE is listed as not available for the baseline, so the calibration comparison is one-sided.

Question 3

What happens when I point it at a domain it was never trained on?

Accepted Answer

It gets worse than the zero-shot alternative, and the README says so in a section it labels an honest boundary. On a deliberate out-of-distribution workload of customer support domains it never saw in training, the head-trained model scored 31.7% against 40.0% for the zero-shot generative baseline. Head training specializes the model for the data you gave it rather than widening it, which is why the pipeline isolates OOD sources and reports them separately instead of folding them into the headline number.

Question 4

What does head training actually cost me in time and memory?

Accepted Answer

Minutes on an Apple Silicon laptop, per the README, because the backbone never updates and the candidate features are forwarded once and cached. The head itself is about 0.8M parameters, a shared scorer plus set attention trained with distribution losses. The documented pipeline is build-data, posttrain with a mode and a steps count and a max-memory-gb flag, then calibrate on dev, then compare. The README does not publish training cost, GPU-hours, or a wall-clock breakdown.

Question 5

What is the full pipeline to build my own decision head?

Accepted Answer

Four CLI steps after uv venv and uv pip install. minojev build-data converts public sources into train, dev, test and OOD splits; minojev posttrain freezes the backbone and trains the head with a mode, a steps count, an inference dtype and a max-memory-gb flag; minojev calibrate fits the per-primitive temperature on dev outcomes; minojev compare benchmarks against a token-generating baseline on the same backbone.

Question 6

Should I serve the head-trained model or plain logits?

Accepted Answer

Serve the calibrated head-trained checkpoint if you trained one, exposed with minojev serve --checkpoint runs/general-cal --port 8000 and posted to /score. The zero-training alternative reads declared option logits directly at letter slots through minojev.logits, including a two-stage path for more than 16 candidates; it scored 88.3% on the bundled suite against 95.8% for the head-trained model.

Question 7

What do the selective-confidence numbers say about accept and escalate gates?

Accepted Answer

The README's separate 200-request calibrated run reports 97.5% accuracy and ECE 0.014 with mean confidence 0.963, and at confidence at or above 0.9 it covers 89.5% of requests at 98.9% accuracy, which it frames as usable for accept and escalate policies. Choice accepts 2 to 255 candidates and score 2 to 10 ordered levels, and question ids are never placed in the model input.

Minojev

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community