Which languages does JevAlt cover?

Three: English with 1,277 test rows, Turkish with 1,019 and German with 820, each row holding a situation, typed questions and probability targets.

How well does Deem-4B perform in English?

94.7% on held-out English decisions against Kev-4B's 84.7%. The card notes test and training rows share a data pipeline, so this measures JevAlt's target task rather than general ability.

What do Karar-4B and Wähler-4B score?

Karar-4B reaches 96.8% on Turkish against Kev-4B's 87.1%, and Wähler-4B reaches 92.0% on German against Kev-4B's 81.1%; Kev-4B leads Wähler-4B on GermEval 2017 and 10kGNAD.

Which failures did the live testing capture?

The results/tested folder holds 390 live requests from 4 October 2026 including misses: Deem-4B 122 of 130, Karar-4B 113 of 130 and Wähler-4B 122 of 130, separate from the held-out scores.

How is the split field meant to be read?

Each row's split field is a placeholder that reads train in every file; the file a row lives in, such as data/en/test.jsonl, is its real split.

What licence applies to the rows?

Written rows are Apache-2.0 while converted records retain source terms: MASSIVE CC BY 4.0 with Amazon attribution, Open-Jev CC0, PAWS-X free for any use with Google credited, and typed-decisions Apache-2.0; each row has its own license field.

CommunityJev Benchmarks & Evaluations

JevAlt Benchmark

Q: Is there overlap between test and training rows?

There is no English state-text overlap, but 14 Turkish and 11 German test states repeat in training through MASSIVE or PAWS-X with different questions; related rows stay in one split by group hash and test files were frozen before training.

English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.

See highlights

Open huggingface.co

Format: Docs
Published by: Community
Author: Mert Kaya
Last verified: Oct 2026

Tagscommunity benchmark evaluation multilingual routing

Works forJev for Evaluation Jev for Classification Jev for Routing

Used inJev for Software & DevTools

Languages: Python
Capabilities: Open source

Highlights

3,116 test rows: 1,277 English, 1,019 Turkish and 820 German, each with a situation, typed questions and probability targets.
Headline results: Deem-4B 94.7% vs Kev-4B 84.7% English, Karar-4B 96.8% vs 87.1% Turkish, Wähler-4B 92.0% vs 81.1% German.
Live testing on 4 October 2026 captured 390 requests including misses: Deem-4B 122/130, Karar-4B 113/130, Wähler-4B 122/130.
Rows carry per-record licence fields: written rows are Apache-2.0, converted records keep MASSIVE, Open-Jev, PAWS-X and typed-decisions terms.

Watch out

Dataset-level licence is 'other' because row terms differ. The card warns the benchmark is not an independent general ranking: test and training rows share a pipeline, each row's split field reads train and the file path is the real split, and 14 Turkish plus 11 German test states repeat in training through MASSIVE or PAWS-X with different questions.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.

FacebookClinicnotes

YouTube: Jev explained for routing and judging, with a comparison against LLMs.

Hacker News: Vercel on using Jev for defined decisions and GPT-6 Astra when the job also has to write or use tools.

flashbrew4 pointsTypeSafe AI Jev vs. GPT-6 AstraRead the thread on Hacker News

Hacker News: ASTGL replay chose local Laya routing over hosted Jev on one Mac Studio setup.

Jmeg8r15 pointsDoes an open-weight decision model beat a hosted one? Jev vs. LayaRead the thread on Hacker News

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Jev-AR Bench

Arabic intent-routing benchmarks split across MSA, Emirati, Saudi and code-switched slices: 4,974 held-out queries over 60 routes plus a 20-question unseen-domain RAG test, CC BY 4.0.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

EdgeIntent v1

A benchmark of natural-language requests to edge services paired with typed intent contracts: 33 conditions of 300 test cases each, from a UTS study on Jev decision models for edge orchestration.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

What the Jev

A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.

Jev Benchmarks & Evaluations#community#python#evals

Community

keta1930/what-the-jevPython

Open-Jev v1.1 Data

326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

JevEmbed Data

1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

Jev Safety Bench

Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.

Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation

Community

From the Jev guides

Original write-ups on what this entry is for.

Confidence Gates in Production: Thresholds That Match the Cost of Being Wrong

Sep 18, 2026

Confidence is not accuracy, and a single global threshold is rarely the right policy. A practical guide to per-action gates, measured thresholds, and the logs you will want later.

Jev Practices & Patterns#confidence#routing#evaluation

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai