CommunityJev Guides & Articles
What Jev can do inside an agent harness
A black-box pass over ten public datasets with jev-1.13.0: injection detection, reranking, intent, tool routing, and three tasks the author says do not work.
- Format
- Article
- Published by
- Community
- Author
- Aitejiu
- Last verified
- Sep 2026
Highlights
- About 22,500 calls, 52.2 million input tokens, $2.19. Code is in Aitejiu/jev-harness-lab.
- InjecAgent, 1,105 items, threshold 0.10: precision and recall 100%, zero false positives.
- Banking77 80.3% over 77 classes. Model-difficulty routing 51%. Non-English recall@1 48.9% versus English 61.5%.
- Choice stops at 255 options. Splitting a shell-risk judgment dropped false positives from 14.5% to 1.8%.
Watch out
Posted 21 September 2026. Some sets are samples or hand-built, including 130 shell commands. One model version.
Questions about this entry
Short answers drawn from the linked project or page.
It is one point on a published sweep, and the lower threshold is doing real work. At 0.50 the same run gives 100% precision with 89.8% recall and a benign mean score of 0.03; at 0.10 recall reaches 100% with no benign false positives. Certainty gating, proxied by max(p, 1-p), auto-handles 41.1% of traffic at full accuracy.
Mostly the second. Tool-relevance detection on 1,140 BFCL items reaches 77.0% only after iteration, and the remaining gap is prompt and structure work rather than a model change. Tool catalog routing on MetaTool is much cleaner at 96.5%, though that is with five similar distractors.
The non-English ones. Korean recall@1 came in at 48.9% against 61.5% for English, which the author attributes to English-primary training, and it is listed among the three things that do not work. Two more negatives: model-difficulty routing at 51% and trajectory failure attribution at AUROC 0.560, both needing to reason about another model's behaviour.
Into the question design, not the model. Splitting one fuzzy judgment into orthogonal questions composed in code took false positives from 14.5% to 1.8%, and letting choice compete before noul verifies moved multi-skill routing from 9.0% to 81.0% recall@1. Related limits: choice caps at 255 options, and state plus questions share roughly 32k tokens.
A black-box engineering evaluation: 10 public datasets, about 22,500 API calls, 52.2M input tokens, and $2.19 total, run against jev-1.13.0, with code and raw results in the jev-harness-lab repository.
For fuzzy semantic judgments, orthogonal decomposition plus code composition beats single-question prompting (shell-gate false positives 14.5% to 1.8%); for multi-label decisions, let choice compete first, then let noul verify (multi-skill routing 9.0% to 81.0% R@1).
choice caps at 255 options, state plus questions share about 32k tokens, input is text only, and it is English-primary, with CJK accuracy dropping. Input tokens are billed; output is free.
More Jev resources like this
Archestra labels 100 production Claude Code tool calls and compares Jev with a constant benign baseline, Sonnet 5, Laya, and several open decoders.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
Hacker News link to Archestra's write-up comparing Jev with a constant benign baseline on production Claude Code tool calls.
Jev Guides & ArticlesPost#community#agents#evaluation
Community
3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonLatent Space's AINews roundup of launch day: the Jev thread atop Hacker News, the RLCD decision-model framing, and community readings that compare it to DSPy-style typed signatures for routing and scoring.
Jev Guides & ArticlesArticle#community#article#models
Community
DataCamp's explainer walks the eval tables behind System One models, from 67.8% workflow accuracy and 0% structured-output errors to the $0.042 per million token price and how to call the API.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
744GitHub stars
A curated list of public projects and practices built on Jev: 212 entries across 13 categories, from classification and routing to agent decisions, calibration, infra/SDKs, and finance, with explicit inclusion rules.
Jev Guides & Articles#community#awesome-list#agents
Community
v-modal/awesome-jev-tools
v-modal/awesome-jev-toolsFrom the Jev guides
Original write-ups on what this entry is for.
The typed-answer shape removes a category of data risk, not all of it. Zero data retention per surface, the legal set to read, and the controls that remain yours to run.
Jev Guides & Articles#privacy#compliance#legal
Read article
TypeSafe serves Jev from one API. A second tier of independent sites wraps it with their own keys, credits, and playgrounds. How to read the disclaimers before you paste a key.
Jev Guides & Articles#ecosystem#trust#hosting
Read article
The first ten days produced three receptions that rarely meet: a Product Hunt launch, a 620-point Hacker News thread, and a r/LocalLLaMA that doubted the framing. Reading them together.
Jev Guides & Articles#launch#community#reception
Read article
From the Jev community
Posts from builders shipping with Jev right now.