CommunityJev Practices & Patterns
Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev) (arXiv 2609.24052)
A system that codes Texas crash narratives as gated, typed Jev decisions returning probabilities and no text: a screen covered 499,500 narratives and 195,857 were coded with a 27-question schema. The typed model attains F1 of 0.908 against human labels, and recalibration cuts calibration error 3.3-fold.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Coding cost is governed by schema size rather than narrative length.
- A screen covered 499,500 Texas narratives and 195,857 were coded with a 27-question schema.
- The typed model attains F1 of 0.908 against 2,416 blinded human judgments.
- One frontier model gains 0.059 over the typed model and the other is indistinguishable from it.
- Recalibration on the same labels reduces calibration error by a factor of 3.3.
Watch out
arXiv preprint posted September 21, 2026 (cs.CL) by Amir Rafe and Subasish Das. Calibration varies by model rather than by paradigm, so each model must be audited; agreement with coded fields understates fidelity to the narrative by a median of 0.26 in kappa, and the added-variable impact is an estimate built from the studied records.
Questions about this entry
Short answers drawn from the linked project or page.
Frontier large language models are costly at that scale, their generated text cannot be verified, and no rule says how much output a human must check.
As gated, typed decisions answered by Jev, a System One model that returns probabilities over analyst-defined options and generates no text.
A screen covered 499,500 Texas narratives and 195,857 were coded with a 27-question schema.
Cost is governed by schema size rather than narrative length, according to the abstract.
Against human labels it attains an F1 of 0.908; one frontier model gains 0.059 and the other is indistinguishable from it.
Recalibration on the same labels reduces calibration error by a factor of 3.3, and the audit uses 2,416 blinded human judgments drawn under a stated sampling design.
Agreement with coded fields understates fidelity to the narrative by a median of 0.26 in kappa, and adding the calibrated variables raises the injury and fatal crashes attributed to nine factors by 10,747 per year.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: A 26-sheet plan set in 2.9 seconds
X: Which exam questions come back
X: 900 images in 40 seconds
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A theoretical study of recalibrating many related yes/no questions that each have few labels, motivated by decision models such as Jev. It derives minimax compound-risk rates and shows pooled isotonic fitting with one per-question correction achieves them, while tuning questions separately loses a factor.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A position and case-study paper on using System One decision models like Jev and the open-source Laya as decision layers in LLM-driven penetration-testing harnesses. It names four decision points, runs an exploratory 13-vulnerability NeuroSploit comparison, and proposes Rave, an unevaluated model.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Documents a failure mode in letter-bound remote sensing classification: 97.71% OA on UCM in vocabulary but 7.35%–96.40% cross-dataset accuracy and inverted confidence (AUROC 0.38). RSAJEV replaces the letter readout with training-free prototype retrieval, reaching 89.43% on the UCM cross-dataset gate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
CalOn is a unified 0.6B-parameter encoder that jointly encodes text and type definitions to extract entities, relations, and event triggers and arguments across datasets and schemas. Across 40 benchmarks it outperforms the evaluated baselines on most and leads all reported relation extraction benchmarks.
Jev Models & ReimplementationsArticle#community#research#arxiv
Community
4GitHub stars
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Glax147/jev_ordinal_scale_biaPython
Glax147/jev_ordinal_scale_biaPython3GitHub stars
PACT turns contrastive training pairs into four loss terms for single-token typed-decision models. On a frozen 324-item holdout it matches the published recipe's accuracy (84.6% versus 85.2%) while cutting answer flips under relabelling from 13.8% to 9.8%.
Jev Practices & PatternsArticle#community#research#arxiv
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
From the Jev community
Posts from builders shipping with Jev right now.