CommunityJev Benchmarks & Evaluations
Jev-AR Bench
Arabic intent-routing benchmarks split across MSA, Emirati, Saudi and code-switched slices: 4,974 held-out queries over 60 routes plus a 20-question unseen-domain RAG test, CC BY 4.0.
- Format
- Docs
- Published by
- Community
- Author
- atmaneayoub
- Last verified
- Oct 2026
Used inJev for Software & DevTools
- Languages
- Python
- Capabilities
- Open source
Highlights
- gov_test holds 4,974 held-out questions over 58 intents plus 2 escape options, with 519 adversarial cases and 6 routes held out from training.
- Leaderboard: Jev-AR at 322M scores 83.4% (95% CI 81.2–85.4) against Qwen3.8-27B's 90.0% and hosted Jev 1.13's 86.2%.
- health_rag covers an unseen app with 6 routes and 20 questions; Jev-AR answers 38 of 40 with no answer changes when the route list is reversed.
- Questions were written by DeepSeek-V4.1-Flash and frozen before any model was trained, split evenly across MSA, Emirati, Saudi and mixed Arabic–English slices.
- An independent DeepSeek label check agrees with the gold label on 94.2% of gov_test, leaving a cleaner 4,686-question subset.
Watch out
CC BY 4.0. The card calls the benchmark synthetic: every question was LLM-written and none was validated by native speakers, some adversarial labels are debatable, and with only 20 health_rag questions a 2-point difference out of 40 is not significant. It is not affiliated with any government or company.
Questions about this entry
Short answers drawn from the linked project or page.
Routing Arabic messages to the right destination: a 4,974-question Gulf-Arabic government-services test over 60 routes, and a 20-question unseen-domain test over a health-insurance RAG app's 6 routes.
On the 60-route gov_test Jev-AR scores 83.4% with an 81.2–85.4 confidence interval against Qwen3.8-27B's 90.0% and hosted Jev 1.13's 86.2%; on health_rag it answers 38 of 40 with no changes when the list is reversed.
Four, split evenly: MSA, Emirati, Saudi and mixed Arabic–English, with country labels uae and ksa and an adv_type field distinguishing adversarial categories.
full-60 uses all 60 options in the canonical Arabic name order from gov_test/intents.json, and sub-20 uses the gold option, the 2 escape options and random options up to 20.
gov_test was written by DeepSeek-V4.1-Flash from realistic scenarios and frozen before any model was trained; health_rag was written with Claude and reviewed by the author.
An independent pass by DeepSeek, the generator's own family, agrees with the gold label on 94.2% of gov_test; the card suggests keeping the 4,686 agreeing questions as a cleaner subset and notes some adversarial labels are debatable.
CC BY 4.0 under the repository LICENSE; gov_test was generated with DeepSeek-V4.1-Flash under MIT with API terms allowing it, and the benchmarks are not affiliated with any government or company.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
YouTube: Jev explained for routing and judging, with a comparison against LLMs.
Hacker News: Vercel on using Jev for defined decisions and GPT-6 Astra when the job also has to write or use tools.
flashbrew4 pointsTypeSafe AI Jev vs. GPT-6 AstraRead the thread on Hacker NewsHacker News: ASTGL replay chose local Laya routing over hosted Jev on one Mac Studio setup.
Jmeg8r15 pointsDoes an open-weight decision model beat a hosted one? Jev vs. LayaRead the thread on Hacker NewsMore Jev resources like this
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A benchmark of natural-language requests to edge services paired with typed intent contracts: 33 conditions of 300 test cases each, from a UTS study on Jev decision models for edge orchestration.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPython3GitHub stars
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPython3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonReal human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
Confidence is not accuracy, and a single global threshold is rarely the right policy. A practical guide to per-action gates, measured thresholds, and the logs you will want later.
Jev Practices & Patterns#confidence#routing#evaluation
Read article
From the Jev community
Posts from builders shipping with Jev right now.