Evaluation & TrainingStrictly proper scoring rule

Proper scoring rule

A reward or error measure that is optimized only by reporting true probabilities — the training signal behind RLCD.

Category: Evaluation & Training
Also known as: Strictly proper scoring rule
Related terms: 5
Directory entries: 66
Docs: —
Added: 2026-10-10

Definition

RLCD trains Laya against strictly proper scoring rules: log and spherical scores, plus a ranked probability score for ordinal questions, with REINFORCE updates and a group-mean baseline. Because only honest probabilities maximize expected reward, calibration is the training objective rather than a post-hoc patch.

The same family names the evaluation metrics. Brier score and log loss are proper rules, which is why they appear in both training tables and benchmark reports, and why a model tuned for accuracy alone can top a leaderboard while losing on Brier score and ECE.

Tagscalibrationevaluationtraining

Related terms

Definitions that connect to this one.

RLCDTypeSafe's post-training method: reinforcement learning that optimizes for calibrated decisions and probabilities instead of generated text.Evaluation & Training Brier scoreA score for a full probability vector against one observed label. Lower is better, and a confident wrong answer costs more than a hedge.Evaluation & Training Expected calibration errorThe average gap between predicted confidence and observed accuracy, grouped into bins. Lower is better, and the number depends on the bins.Evaluation & Training CalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & Confidence Temperature scalingDividing a model's logits by one fitted number per setting to bring its probabilities closer to observed frequencies.Evaluation & Training

From the Jev directory

Laya Typed-Decisions

The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.

Jev Models & Reimplementations#community#open-models#local

Community

Laya on Hugging Face

The Hugging Face hub for Laya's open System 1 checkpoints: the 421M English model at the repo root, a 322M multilingual build in a subfolder, and the typed-decisions fine-tune, all Apache-2.0.

Jev Models & Reimplementations#community#open-models#local

Community

Decider Chat Gemma-4-31B

An untrained readout: stock Gemma-4-31B weights plus a decider config that reads option letters at the answer slot, scored #2 of 70 on Decision Index v0.2.1 at 57.33 against Jev 1.13.0's 57.91.

Jev Models & Reimplementations#community#open-models#local

Community

Typed Decisions Benchmark

The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.

Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation

Community

Tasksource Jev Typed-Decisions

2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.

Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration

Community

Procedural Typed Decisions

572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.

Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration

Community

60 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai