Docs
CommunityJev Benchmarks & Evaluations
6G-JEV (RANIntent v1)
RANIntent v1: an O-RAN intent-interpretation benchmark from UTS with 16 telemetry conditions, where an interpreter returns one A1 policy with 9 fields; 300 test and 60 dev cases per condition.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- UTSCybeR
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 16 configs: 3, 7, 21 or 57 cells crossed with fresh, stale, noisy and contradictory telemetry; each has 300 test and 60 dev cases.
- Interpreter output is one A1 policy of type 20001 with 9 fields, from action values such as prioritise and cap_share to class, scope and priority.
- 150 test cases per condition are state-dependent: the scope is most_loaded_cells or worst_edge_cells and the telemetry picks the target cluster.
- Paper: arXiv 2609.23136, Intent Interpretation at RIC Timescales, by Delong Li and colleagues at the University of Technology Sydney.
- Scores cover target_cluster accuracy, actuated-field match, full-policy match, schema validity and unsafe policies; CC BY 4.0.
Quickstart
python
from datasets import load_dataset
ds = load_dataset("OniReimu/6G-JEV", "c21_fresh")Watch out
Card reviewed 2026-10-17; the dataset accompanies the authors' own paper (arXiv 2609.23136) and its comparisons of Jev decision models against large language models are their reported results, not independent evaluations. Dev cases are only for prompt checks, and telemetry token counts grow steeply with cell count.
Questions about this entry
Short answers drawn from the linked project or page.
A benchmark of natural-language intents for an O-RAN network: each intent is paired with the A1 policy it expresses and a KPM telemetry table carrying the network state, built for the 6G-JEV paper.
One policy of A1 policy type 20001 with 9 fields — action, class, scope, target_cluster, priority, prb_share, edge_site, latency_target and duration — each set to one allowed value from the task schema.
3, 7, 21 or 57 cells crossed with fresh, stale, noisy or contradictory telemetry, each a dataset config with a 300-case test split and a 60-case dev split sharing the same intents and labels.
On accuracy for target_cluster, the match of actuated fields (action, class, scope, priority), full-policy match, schema validity and unsafe policies, with reading rules given to every interpreter and the verifier.
Half of every test split names no cluster: the scope is most_loaded_cells or worst_edge_cells and the target cluster is whichever contains the extreme cell of the telemetry, so the model must read the table.
Through the datasets library by config name, for example load_dataset("OniReimu/6G-JEV", "c21_fresh"), with test and dev splits; code, harness and reproduction instructions live at github.com/OniReimu/6G-JEV.
CC BY 4.0; apart from the runs directory the repository layout matches the GitHub repository, so files can be dropped into a clone of the code for reproduction.
More Jev resources like this
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPythonA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonEvaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonRun outputs behind the JevGPT paper's tables: 830 BrowseComp-Plus records at 90.48% accuracy and 105 NeuCLIRBench topics at 0.5390 nDCG@20, produced by a GPT-5.5 System Two paired with Jev-1.13.0 as System One. Per-item cost, latency and encrypted trajectory files are included. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
xiq's JEV-format view of OpenJev-Data: every browser step re-expressed as state, questions and gold across eight configs covering Mind2Web, NNetNav, WebChain, two teacher runs and on-policy GLM, plus a frozen 2,346-step bench and 153 live tasks. Licences are mixed and per-row terms apply.
Jev Benchmarks & EvaluationsDocs#community#evaluation#browser-agent
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.