Jev Calibration

The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.

An untrained readout: stock Gemma-4-31B weights plus a decider config that reads option letters at the answer slot, scored #2 of 70 on Decision Index v0.2.1 at 57.33 against Jev 1.13.0's 57.91.

The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.

2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.

572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.

A native Brazilian Portuguese typed-decisions benchmark: seven choice and noul tasks over legislative, legal, scientific, and fact-checking text, with a tuned 4B model at 72.9% mean.

An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.

The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.

Apache-2.0 ModernBERT fine-tune that answers noul, choice, score and multi-label questions with a probability for every option. It serves Jev's POST /v1/systemone locally so TypeSafe's Python SDK runs unchanged, reporting 0.026 expected calibration error on its test split.

An Apache-2.0 0.8B decision model that returns calibrated probabilities for yes/no, choice and score questions on CPU. Distributed as GGUF for llama.cpp and served behind a Jev-style /v1/systemone API, so existing Jev clients mostly need only a new base URL.

The first Chinese-language calibration benchmark for Jev-class System One decision models: 378 items and 467 questions across voice routing, e-commerce service and content moderation, with accuracy, ECE, Brier, NLL, AUROC, option-order and zh-CN/zh-TW robustness metrics.

Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.

Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.

Reproducible recruiting cookbook that screens one resume against a job description in a single parallel Jev request of five Noul evidence gates, four Score dimensions and one background-routing Choice, with criteria iterated v1 to v3 against positive, negative and trap resumes.

A local Jev-compatible server that reads Choice, Score and Noul answers from the logits of any MLX checkpoint or OpenAI-compatible endpoint with no training, serving POST /v1/systemone as a drop-in for the official TypeSafe SDK.

An open recipe for a 27B multimodal decision model trained with full-weight SFT on 73,000 examples over one H200, serving choice, noul and score on /v1/systemone with per-checkpoint provenance and released calibration plots.

Open recipe for calibrated decision models: an LLM teacher writes training data from a one-sentence task description, a LoRA plus decision head on Qwen3-4B-Base answers Choice, Score and Noul questions with calibrated probabilities, evaluated next to Jev on identical items.

Head-to-head benchmark of fast typed classifiers, TypeSafe Jev, OpenAI Decisions and Cloudflare Clef, against Claude models on a frozen, labeled corpus of 2,321 cases across 31 tasks, measuring accuracy, calibration, p95/p99 latency and registry-priced cost.

Dependency-free TypeScript SDK that turns Jev Noul, Choice and Score answers into named decisions: enter/exit thresholds with hysteresis, nested decision trees settled in one call, a JSONL journal, threshold replay without inference and Brier calibration.

Node.js CLI that checks whether published Jev outputs were actually produced by Jev: it recomputes the Yurin confidence identity and the expected-score formula over a repository's artifacts and flags fixtures that violate them. No API calls, no dependencies, Node 18+.

Ruby gem that makes System One measurement a language primitive: psi prepares a state, judge/choose/score return calibrated distributions, and ? collapses them for control flow. Ships TypeSafe, protocol (/v1/systemone) and CUA providers behind one interface with its own spec suite.

Python library that records each Jev-class decision as the full distribution under a sha256 key of state, question, provider and model, then applies a separate policy. Thresholds, review bands, routing and budgets can change across the recorded history without new model calls.

A preregistered Quick, Draw! study: 400 SVG-sketch classifications with Jev versus Sonnet on the same text, plus base64 and image baselines.

An independent poker eval against TexasSolver: Jev matches solver top actions on easy spots but shoves the nuts into traps and needs explicit hand-rank text to fold to shown flushes.

Thread on Kartik Pansuriya's essay arguing calibrated probabilities matter more than headline accuracy for production classifiers.

A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.

A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.

Sixteen true-or-false statements you answer, then Jev answers with a Noul probability. Keeping only its confident calls is the point of the game.

Alex Molas argues that a probability fit on TypeSafe's data is not calibrated on yours, so Jev's numbers should be treated as scores.

The Hacker News thread on Alex Molas's calibration argument, including a commenter who reran the fair-coin and fair-die prompts.

Nokia Applied Research's package that turns any open LLM into a Jev-style decision model: typed questions are read from a single prefill, and the calibration fixes need no fine-tuning.

A roughly 10,000-call black-box investigation that reconstructs Jev's likely architecture: a shared state prefix, isolated question branches, and direct probability readouts.

A critical technical read of Jev's launch claims that digs into the calibration and eval numbers and asks what the architecture keeps secret.

A skeptical Japanese replication that approximates Jev's speed by reading first-token logits from small LLMs, with Gemma3 and a Mario benchmark.

Published evaluations of four automation workflows, security incidents, agent trace observability, invoice processing, and customer service, comparing Jev and frontier LLMs as structured workflows versus single prompts.

Browser playground in the TypeSafe console for trying System One: paste state, define typed questions, and inspect answers with probabilities, confidence, latency, and usage.

Side-by-side benchmark of TypeSafe Jev, Qwen 3.8 27B on Cerebras, and a local Needle 3 across seven synthetic workloads, recording validated outputs, mistakes, latency, tokens, and estimated cost. Raw exports and per-scene limitations are published.

CLI that fits a per-question confidence threshold to a target accuracy on your own labeled data, verifies it on a held-out split, estimates how much traffic still needs an LLM, and re-checks locked thresholds in CI. It publishes no Jev results of its own.

An agent skill for deciding where typed judgments belong beside code, policy, and generation, with Jev examples for question design and abstention plus an offline probability and threshold evaluator.

Blind security benchmarks for Jev on public corpora: 662 prompt-injection messages and 200 matched vulnerable-code pairs, with raw per-sample results and a TUI dashboard.

An independent open reproduction of the System One model class: Qwen3.5-based 2B and 35B mixture-of-experts models that return typed Choice, Score, and Noul probabilities in one forward pass, with nothing distilled from Jev.

An independent baseline that reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass, reproducing Jev's interface pattern with open models rather than Jev's undisclosed model or training.

Apache-licensed, locally runnable Jev-style decision models (0.8B, 4B, 9B on Qwen3.5) with released weights, training code, a System One-compatible server, frozen eval suites, and a playground.

An independent calibration study of Jev over three public benchmarks and 900 rule-generated support tickets, publishing every raw Gateway response and the quantization limits of returned probabilities.

A public benchmark comparing Jev with Claude Haiku 4.5 on whether an email agent should click a link, over 2,000 PhishNChips emails, with calibration, latency, cost, and five signal questions.

An Apache-licensed Python library that reads option-token probabilities from open-weight models through Transformers or vLLM, giving Jev-style typed decisions in one forward pass without an HTTP server.

A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.

An independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.

A ModernBERT-based open decision model that evaluates Choice, Score, and Noul schemas in one non-autoregressive pass; Verdict 2.0 reports 77.10% accuracy on 2,000 held-out enterprise decisions.

Unofficial companion to dspy-typesafeify that records and replays TypeSafe calls to measure calibration, selective risk, confidence-gated abstention, latency, tokens and modeled cost for Jev decisions.

Independent measurement of whether ORDER BY over a Jev probability is defensible, pre-registering gates on pairwise inversion, Score ordinality, calibration, and negation and paraphrase invariants.

Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.

A pre-registered, independently rescored audit of Jev's calibration and type safety: calibrated on CLINC150 (ECE 0.0204), systematically overconfident on Banking77 (ECE 0.0936), and zero type errors across 8,576 responses.

A from-scratch one-pass option scorer that maps a context and a changing list of text options to one probability per option, adding a hashed n-gram encoder, rival-aware attention, a gated head, and temperature scaling; JSONL data is drop-in compatible with jevlike.

One-pass option scoring with a local Gemma 3 4B on Apple silicon via MLX: it prefills the context once, expands that KV cache across the option batch, and scores every option in a single padded forward pass with no decoding.

An encoder-side reproduction of Jev built on a retrained GLiClass ModernBERT base (151M): it evaluates multiple typed questions in one non-autoregressive forward pass, returning choices, ordinal scores, and probabilities under 35 ms.

An open re-creation of Jev's decision model on an open-weights Qwen3.5-4B: it answers a list of typed questions in one forward pass over POST /v1/systemone in about 200 ms, never writing free text.

An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.

A bilingual field map of where Jev fits and where it fails, separating the author's raw API suites from cited third-party results and TypeSafe's own claims.

An open, local-first alternative to Jev that reproduces the Choice, Score, and Noul interface on commodity zero-shot NLI models with temperature scaling and conformal abstention, and runs offline with no API key.

An independent measurement spanning 5,477 test rows and 34.1M input tokens, comparing one direct Jev question per row with 12-14 scored dimensions fitted to local labels.

Sixteen thousand calls against gpt-5.4-mini and gpt-5.6-luna on four public datasets plus production pipeline decisions, with a threshold procedure and an explicit filter-not-replacement verdict.

An independent trial running Jev on 24 Norwegian resource-tax hearing responses with eleven typed questions each, then comparing agreement, calibration, latency and cost against DeepSeek V4.1 Flash.

Notes from a technical analysis that inferred Jev's internals from roughly 1,000 API calls, probing how probabilities and confidence behave.

A comparative review of Jev and the open-source Laya encoder: interfaces, deployment, disclosed internals, calibration claims, and the limits of agreement-based workflow evals.