Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
CommunityJev Benchmarks & Evaluations

Zh Decision Bench

The first Chinese-language calibration benchmark for Jev-class System One decision models: 378 items and 467 questions across voice routing, e-commerce service and content moderation, with accuracy, ECE, Brier, NLL, AUROC, option-order and zh-CN/zh-TW robustness metrics.

See highlightsOpen github.com
Stars
3
Language
Python
License
APACHE-2.0
Published by
Community
Last verified
Oct 2026

About this project

A calibration benchmark for "System One" decision models on Chinese tasks — measuring not just whether the model picks the right answer, but whether the probabilities it reports can be trusted. Around the decision-model category Jev (TypeSafe AI, Sept 2026) opened up — non-generative, single forward pass, typed decisions with calibrated probabilities — every public evaluation so far is English-only (the Just Ask Jev paper explicitly lists other languages as future work). This repo fills the Chinese gap.

Quoted from the project's README. Read the full README.

Tagscommunitybenchmarkcalibrationchinese
Works forJev for EvaluationJev for ClassificationJev for Moderation
Used inJev for Customer SupportJev for E-commerce & Retail
Languages
Python
Capabilities
Open source

Highlights

  • 378 items / 467 questions across three question primitives: choice, score (ordinal) and noul (binary).
  • Jev (jev-latest) reached 0.960 accuracy / 0.035 ECE on 323 voice-routing items; NeoHorse-Jev-4B reached 0.950 / 0.031 and won CS routing and escalate outright.
  • Five models evaluated: Jev API, NeoHorse-Jev-4B, Laya multilingual 322M, Laya English 421M and a Qwen3.5-2B logit-probe baseline.
  • All 2,335 raw model predictions ship in results/raw/, and the dataset is CC BY 4.0 on Hugging Face.
  • The eval set ships in Laya v0.3.21 and includes option-order flips and Simplified/Traditional Chinese robustness tests.

Watch out

Apache-2.0 code with a CC BY 4.0 dataset. Business-scenario groups are small (n=15-34) with wide confidence intervals, and synthetic items were LLM-drafted and adjudicated by a single person with a public review log.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

jev vs. open alternatives
7GitHub stars
Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Communityrun-llama/jev_vs_ossJupyter Notebook
Jev Synthetic Survey
2GitHub stars
Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.
Jev Benchmarks & Evaluations#community#benchmark#calibration
Communityjjd-lab/jev-synthetic-surveyPython
jev-acento
0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Communitymarcosmartinez/jev-acentoPython
classifier-bench
0GitHub stars
Head-to-head benchmark of fast typed classifiers, TypeSafe Jev, OpenAI Decisions and Cloudflare Clef, against Claude models on a frozen, labeled corpus of 2,321 cases across 31 tasks, measuring accuracy, calibration, p95/p99 latency and registry-priced cost.
Jev Benchmarks & Evaluations#community#benchmarking#calibration
CommunityCMaintz/classifier-benchPython
Gutsy
77GitHub stars
An Apache-2.0 0.8B decision model that returns calibrated probabilities for yes/no, choice and score questions on CPU. Distributed as GGUF for llama.cpp and served behind a Jev-style /v1/systemone API, so existing Jev clients mostly need only a new base URL.
Jev Models & Reimplementations#community#gguf#local-inference
Communitykouhxp/gutsyHTML
Luce
8GitHub stars
Open recipe for calibrated decision models: an LLM teacher writes training data from a one-sentence task description, a LoRA plus decision head on Qwen3-4B-Base answers Choice, Score and Noul questions with calibrated probabilities, evaluated next to Jev on identical items.
Jev Models & Reimplementations#community#calibration#training
Communityscienthoon/lucePython
Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai