Article
CommunityJev Benchmarks & Evaluations
Decision Models as Policy Gates for Coding Agents
arXiv paper evaluating 22 hosted decision models and two prompted general models as runtime classifiers for a multi-rule coding-agent policy engine. It reports recall, false-positive, latency, and cost results on a 2,643-call synthetic dataset with a sealed holdout.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates 22 hosted decision models and two prompted general models as classifiers for a multi-rule policy engine.
- Uses a synthetic dataset of 2,643 tool calls labeled blind against 28 rules, with a sealed holdout.
- Explicit boundaries as examples and counterexamples raised recall by 5.5-26.1 percentage points on all 23 available models.
- Jev 1.14's recall rose from 89.3% to 98.2%, and unnecessary interruptions fell from 30.6% to 4.8% of 252 clean actions.
- With examples, Jev 1.14's median latency was 120 ms at $0.46 per thousand requests.
Questions about this entry
Short answers drawn from the linked project or page.
It evaluates 22 hosted decision models and two prompted general models as classifiers for a multi-rule policy engine that gates coding-agent tool calls.
It uses a synthetic dataset of 2,643 tool calls labeled blind against 28 rules, with a sealed holdout for evaluation.
Writing boundaries as examples and counterexamples raised recall by 5.5-26.1 percentage points on all 23 available models and reduced false positives on 21.
Jev 1.14's recall rose from 89.3% to 98.2%, while unnecessary interruptions fell from 30.6% to 4.8% of 252 clean actions.
With examples, its median latency was 120 ms at $0.46 per thousand requests.
On 16 new scenarios written after the initial studies, a follow-up improved recall on 22 of 23 models and reduced unnecessary interruptions on 21.
The authors release the dataset, rule texts and harness used in the evaluation.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Vercel's fx safety reviewer, measured at 18× faster
Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.
X: Decision outputs in Grep
Hacker News: jevmem keeps Claude Code project memory by asking Jev which turns are worth saving.
avinashjetwani61 pointsJevmem – automatic project memory for Claude Code, built on JevRead the thread on Hacker NewsMore Jev resources like this
arXiv paper presenting FlipJev, a fast black-box query-based red-teaming attack that finds short universal adversarial suffixes for Jev. It reports experiments on 12 datasets, 10,240 paired prompt-injection tests, and cross-task suffix transfer of 90-100%.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv paper introducing S²-MAS, a society-scale multi-agent simulation framework built on 500 million synthetic household personas from survey data. Its Genese engine separates decisions from state updates, and sharp-421m-mlx is benchmarked against Jev 1.13 on financial-choice tasks.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Scala 3 benchmark running one Jackson documentation-research task over an MCP catalog through three designs: a direct LLM agent, an LLM with a nested Jev evidence tool, and top-level Jev orchestration, comparing turns, MCP calls, token use and latency.
Jev Benchmarks & Evaluations#benchmark#evaluation#mcp
Communityjamesward/hello-zio-typesafe-ai
A TypeSafe AI Hugging Face dataset of agent trace observability records for evaluation workflows, with cases (111 rows), questions (1.12k rows), and run_results (999 rows) in parquet, including per-provider and consensus decision blocks.
Jev Benchmarks & Evaluations#evaluation#agents#benchmark
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.