CommunityJev Benchmarks & Evaluations
EdgeIntent v1
A benchmark of natural-language requests to edge services paired with typed intent contracts: 33 conditions of 300 test cases each, from a UTS study on Jev decision models for edge orchestration.
- Format
- Docs
- Published by
- Community
- Author
- UTSCybeR
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 33 dataset configs, each with a 300-case test split and a 60-case dev split, under RQ1a, RQ1b, RQ2, RQ3 and RQ4 prefixes.
- An interpreter fills closed-set contract fields: service_type, locality, quality_floor, urgency, retention, energy, redundancy and latency_class.
- Cases are scored by exact match of all fields; the paper also reports per-field accuracy, validity and unsafe placements.
- 23 conditions were generated and verified, and 10 were derived programmatically from verified text.
- runs/ holds about 400 MB with 124,460 interpreter-call records and 50,940 RQ5 arrival outcomes.
Watch out
CC BY 4.0, supporting arXiv:2609.22753 from the University of Technology Sydney group. Label tuples were sampled first and texts written by Gemini-3.8-Flash and Claude-Opus-5.5, kept only when a blind verifier matched every field, so cases are synthetic and verified rather than human-authored; the card's load_dataset example uses the OniReimu namespace, not this repository's owner path.
Questions about this entry
Short answers drawn from the linked project or page.
Whether an interpreter can read a natural-language request to an edge service and fill a typed intent contract, scoring exact match on fields such as service_type, locality, quality_floor and urgency.
33 conditions, each a dataset config with a 300-case test split and a 60-case dev split, grouped under RQ1a input length, RQ1b bundles, RQ2 wording, RQ3 contract size and RQ4 catalog size.
Label tuples were sampled first with balanced marginals, Gemini-3.8-Flash and Claude-Opus-5.5 wrote the request texts, and a blind Claude-Opus-5.5 verifier relabelled each text; a case was kept only when every field matched.
In runs/, about 400 MB with 124,460 interpreter-call records and 50,940 RQ5 arrival outcomes, including returned labels, option probabilities, latency, tokens, billed cost and GPU power traces.
Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration by Delong Li, Xu Wang, Haochen Gong, Rui Lang and Guangsheng Yu at the University of Technology Sydney, arXiv:2609.22753.
Development cases are meant only for prompt checks, classifier training and threshold calibration, not for reporting test accuracy, per the conditions section of the card.
The repository is CC BY 4.0, and the card's load_dataset example loads OniReimu/Edge-Computing-JEV; the IIIT5K-Word images referenced by the RQ5 OCR service are not included and are downloaded by the GitHub script.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
YouTube: Jev explained for routing and judging, with a comparison against LLMs.
Hacker News: Vercel on using Jev for defined decisions and GPT-6 Astra when the job also has to write or use tools.
flashbrew4 pointsTypeSafe AI Jev vs. GPT-6 AstraRead the thread on Hacker NewsHacker News: ASTGL replay chose local Laya routing over hosted Jev on one Mac Studio setup.
Jmeg8r15 pointsDoes an open-weight decision model beat a hosted one? Jev vs. LayaRead the thread on Hacker NewsMore Jev resources like this
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Arabic intent-routing benchmarks split across MSA, Emirati, Saudi and code-switched slices: 4,974 held-out queries over 60 routes plus a 20-question unseen-domain RAG test, CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPython7GitHub stars
A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPython3GitHub stars
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPython3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonFrom the Jev guides
Original write-ups on what this entry is for.
Confidence is not accuracy, and a single global threshold is rarely the right policy. A practical guide to per-action gates, measured thresholds, and the logs you will want later.
Jev Practices & Patterns#confidence#routing#evaluation
Read article
From the Jev community
Posts from builders shipping with Jev right now.