JevTuner

It trains a language model to make closed-set decisions — a state plus a typed question maps to a decision plus a complete probability distribution — instead of generating a long response and then reporting a separate confidence value.

The model's logits are gathered only at the candidate decision slots and softmaxed across just those slots, so every request returns a normalized probability for each allowed decision and the final decision is the argmax.

The tokenizer gains reserved single-token slots named <|decision_0|>, <|decision_1|> and so on; these internal slots are mapped to the request's choice strings, so labels of any length stay single-token decisions.

Tokenized Multi-class Brier Loss, summed over candidates as (p_k minus y_k) squared against a one-hot target. A confidently wrong decision gets a large penalty, and training only needs the final correct choice, not probability targets.

JSON Lines with one decision per row: state, question, a choices array and a zero-based label index. Examples can have different labels and different numbers of choices, up to the --max_choices limit.

Clone the repo, run pip install -r requirements.txt, then run python jevtuner.py train with --model_name Qwen/Qwen3-8B, your JSONL train file, an output directory and optionally --use_peft True. Python 3.10+ and PyTorch 2.3+ are required.

The repository declares no license file, and the README publishes no accuracy or calibration benchmarks. It is an open implementation of the decision-first idea from ConfTuner rather than a hosted service, and its scope statement is truncated in the README.