CommunityJev Benchmarks & Evaluations
Intent Interpretation at RIC Timescales: Jev Decision Models versus Large Language Models in 6G Open RAN (arXiv 2609.23136)
A comparison of Jev-1.13.0 and two other decision models with LLMs for intent interpretation in 6G Open RAN. Jev meets the 1 s near-real-time budget on 99.8% of calls while two hosted LLMs meet it on 17.9% and 0%, with median interpretation taking 0.286 to 2.35 s.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Compares Jev-1.13.0 and two other decision models with LLMs on the RANIntent v1 benchmark.
- Evaluated in closed-loop ns-3 simulation and on a real A1 and E2 path.
- Jev-1.13.0 meets the 1 s near-real-time budget on 99.8% of calls; two hosted LLMs meet it on 17.9% and 0%.
- Median interpretation takes 0.286 to 2.35 s, against under 25 ms for A1 and E2 transfer.
- Two interpreters saturate their queues at 2 intents/s, and no radio penalty of slow interpreters was resolved at the base point.
Watch out
arXiv preprint posted September 19, 2026 and updated October 2, 2026 (cs.NI) by Delong Li and co-authors; 23 pages. The evaluation uses one benchmark, a simulation, and one real path, and the authors report that no hosted LLM showed a resolved increase over Jev-1.13.0 at the base point, so the comparison is bounded to the tested conditions.
Questions about this entry
Short answers drawn from the linked project or page.
Turn intents into A1 policies within the loop of the RAN intelligent controller, where decision models such as Jev-1.13.0 return typed policy fields whereas generative LLMs produce the policy token by token.
Whether the extra delay of LLMs costs control deadlines, RIC capacity, or radio performance.
The RANIntent v1 benchmark, in closed-loop ns-3 simulation and on a real A1 and E2 path.
Jev-1.13.0 meets the 1 s near-real-time budget on 99.8% of calls, while two hosted LLMs meet it on 17.9% and 0%; median interpretation takes 0.286 to 2.35 s against under 25 ms for A1 and E2 transfer.
Ideal enforcement moves the affected-class service-level agreement violation by 3.96 percentage points in the direction each intent requests, against no update at the base point.
No hosted LLM showed a resolved increase over Jev-1.13.0 at that point, and per-second direct control gave no resolved SLA reduction over a numerical xApp.
Two interpreters saturate their queues at 2 intents/s, whereas no radio penalty of slow interpreters was resolved at the base point.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
From the Jev community
Posts from builders shipping with Jev right now.