Docs
CommunityJev Benchmarks & Evaluations
Soro Universal Tool Harness
nisaralll's dataset for training and calibrating System-One pointer routers: 23,192 route, argument-repair and safety rows over six tool-declaration formats, plus zero-shot and human validation splits. Tasks are formatted as pointer queries over explicit option lists. Apache-2.0.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- nisaralll
- Last verified
- Oct 2026
Used inJev for Software & DevTools
- Languages
- Python
- Capabilities
- Open source
Highlights
- 23,192 train rows, plus 2,220 zero-shot rows (tools unseen in training) and 1,200 held-out human queries.
- Six tool declaration formats: MCP JSON-RPC, Agent Skills, OpenAPI v3, Python signatures, GraphQL RPC and mixed catalogs.
- Questions include tool_route with __abstain__, permuted tool_route_perm, arg_mapping, missing_required, is_mutation and intent.
- Formats agent tasks as System-One Pointer Queries over explicit option lists for zero-hallucination routing.
- Apache-2.0; the card links trained Soro-Kev-4B and 0.8B models and a Soro-Boro GitHub repository.
Quickstart
python
from datasets import load_dataset
dataset = load_dataset("nisaralll/soro-universal-tool-harness")
print(dataset["train"][0])Watch out
Card reviewed 2026-10-17; the card reports dataset composition and claims zero-hallucination routing with split-conformal coverage guarantees, but it publishes no benchmark numbers or evaluation results for the trained routers. Row-level label quality, delexicalization effects and the human validation split's provenance are documented only at summary level.
Questions about this entry
Short answers drawn from the linked project or page.
A corpus of routing, argument-repair and safety-classification records for training and calibrating System-One pointer routers. Agent tasks are formatted as pointer queries over explicit option lists rather than token-by-token JSON generation, which the card links to zero-hallucination routing and split-conformal coverage guarantees.
It borrows the decision-model pattern: instead of emitting JSON strings, a router picks a pointer into a supplied candidate list and can abstain with __abstain__ when nothing fits. The card frames this as System-One style routing for tool harnesses, with confidence handled through conformal prediction rather than generated text.
train.jsonl has 23,192 rows (5,853 delexicalized, 25.2%), val_zeroshot.jsonl has 2,220 rows with tools never seen during training, and val_external_human.jsonl has 1,200 held-out human queries from Tau-2 and live MCP benchmarks, none delexicalized.
Six, to prevent syntactic overfitting: Model Context Protocol JSON-RPC schemas, Agent Skills and SKILL.md definitions, OpenAPI v3 REST endpoints, Python function signatures, GraphQL query and mutation RPC schemas, and mixed heterogeneous real-world catalogs.
tool_route picks the correct candidate or abstains, tool_route_perm repeats it with permuted candidates to enforce order-invariance, arg_mapping repairs misnamed argument keys, missing_required detects absent mandatory arguments, is_mutation classifies side effects, and intent marks single, multi or no-tool requests.
The card shows two paths: datasets.load_dataset with the repository id for the standard loader, or hf_hub_download of an individual JSONL such as val_external_human.jsonl followed by json.loads for each line.
nisaralll published it under Apache-2.0, and the card links trained Soro-Kev-4B and 0.8B models. It reports composition and claims about zero-hallucination routing and conformal coverage but publishes no benchmark scores for the trained routers.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: A chat bot with no LLM
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
More Jev resources like this
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonEnglish, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Arabic intent-routing benchmarks split across MSA, Emirati, Saudi and code-switched slices: 4,974 held-out queries over 60 routes plus a 20-question unseen-domain RAG test, CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A benchmark of natural-language requests to edge services paired with typed intent contracts: 33 conditions of 300 test cases each, from a UTS study on Jev decision models for edge orchestration.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
MIT-licensed Python tool router for coding agents that keeps obvious calls on a fast path, asks Jev (default typesafe/jev-1.13 via OpenRouter) for genuinely ambiguous choices, and can run bounded MCTS when multi-step consequences matter. Ships a CLI, MCP server and hook integrations.
Jev Tools & CLIs#community#routing#mcp
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.