Docs
CommunityJev Benchmarks & Evaluations
Xyntetik Typed Decisions Record
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
- Format
- Docs
- Reading time
- 3 min
- Published by
- Community
- Author
- Joakimpalm-Zen
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- TD-001 tested whether a log-loss LoRA of Kvist-14B beats the per-class temperature map by 0.05 nats; it is INCONCLUSIVE.
- TD-002 reran the same adapter on a fresh held-out sample on 2026-10-14/15: also INCONCLUSIVE, its report not yet written.
- Slice 1 has 1,303 questions (464 ledger, 809 synthetic); slice 2b has 8,044 (544 ledger, 3,900 WANLI NLI, 3,600 intent).
- Held out by split group, never by row: 583 of 2,917 ids from a joint split fixed in TD-000 before any data.
- The record holds preregistrations, reports, question files, scored rows, analyses and the harness; no model weights.
Watch out
Card reviewed 2026-10-17; TD-001's adapter was never committed to the record, so the fine-tune cannot be reproduced from the published files directly, and TD-002's report is not yet written. The fetched card text also truncates inside the inventory of the scripts directory, and the INCONCLUSIVE outcomes depend on registered rules whose smallest attainable one-sided p is 1/10,001.
Questions about this entry
Short answers drawn from the linked project or page.
The research record of studies on whether an endpoint's probabilities can be used as decisions: TD-000 measured three models' /v1/decide probabilities, and TD-001 ran a preregistered test of a log-loss LoRA fine-tune of Xyntetik-Kvist-14B against a free per-class temperature map. No model weights are included.
One hypothesis: that the fine-tune reaches a held-out log loss on the ledger class at least 0.05 nats below the base model's per-class temperature map. The registered rule returned INCONCLUSIVE: the paired bootstrap over 32 units covering 174 pooled questions met neither the pass nor the kill bounds.
A dated update (2026-10-15) that reran the same adapter on a fresh held-out sample from 2026-10-14/15. The card states TD-002 is also INCONCLUSIVE, from its analysis file, and that its report is not yet written.
Slice 1 has 1,303 questions: 464 ledger items (does a proposed code edit make the failing tests pass), 809 synthetic Python program states answered by execution, and a 30-question placeholder read as nothing. Slice 2b has 8,044, each item in two or three paraphrased variants.
Studied and fine-tuned: Xyntetik-Kvist-14B, scored from the Q8_0 GGUF and trained from the BF16 release. Measured beside it in TD-000: Muse-Glimmer-30B and Qwen3.5-4B. Comparators in TD-001: the raw base and the base with the per-class temperature map.
One joint split was fixed before any data and held out by split group, never by row: 583 of 2,917 ids. The instrument is Xyntetik Runner v1.1.4 and its decide-calibrate script, both pinned by hash, with contamination audits of slices 2 and 2b.
Apache-2.0. TD-001's LoRA adapter was never committed to the record, so it cannot be reproduced directly from the published files, and the fetched card text truncates inside the inventory of the scripts directory.
More Jev resources like this
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Twelve frozen typed-decision configs from the independent Open-Jev project, publishing controlled and mostly synthetic tasks with probability or ordinal targets: from the 113,568-row default release to the 160,797-row browser and drone expansion. CC0 1.0; English, Chinese and Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.