User repository on GitHub
Paired and self-audited evaluation of System-1 decision models (Laya vs Jev) for LLM agent harnesses
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses (arXiv 2610.02267)
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
- Stars
- 0
- Language
- JavaScript
- License
- MIT
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- 11 agent decision points built from 18 public sources: 7,283 base cases plus 6,640 robustness variants.
- Jev is significantly more accurate on 9 of 11 decision points, by +10.8 to +46.0 points.
- Neither model beats chance on zero-shot model routing; the two tie on RAG relevance gating.
- Laya changes 30% of answers when option order is reversed, and reaches 31% at 50 nearest-neighbour tools.
- Self-audit: an omitted pre-screen cost turned a reported 23.9% saving into an actual 4.3%.
Watch out
arXiv preprint posted October 1, 2026 (cs.AI, cs.CL, cs.CR, cs.LG) by Jiawei Li; 11 pages, 7 figures. The evaluation is paired and self-audited, and the self-audit itself documents three analysis errors and one design confound, so claims should be read alongside the corrected numbers.
Questions about this entry
Short answers drawn from the linked project or page.
Eleven agent decision points built from 18 public sources, covering tasks agent harnesses face repeatedly: which model to call, which tool to use, whether retrieved text is relevant, and whether an input carries an injection.
7,283 base cases plus 6,640 robustness variants, run with byte-identical inputs, paired tests, and cross-hardware and cross-day reproducibility checks. Both the open-weight Laya model and the hosted Jev model saw the same prompts.
Jev was significantly more accurate on 9 of 11 decision points, by +10.8 to +46.0 percentage points. The two models tied on RAG relevance gating, and neither beat chance on zero-shot model routing.
Laya changes 30% of its answers when the option order is reversed. It also degrades sharply with many or similar candidates: 31% at 50 nearest-neighbour tools, versus 98% for Jev on items with a unique correct tool.
Three analysis errors and one design confound distorted headline deployment claims: an omitted pre-screen cost (reported 23.9% saving, actual 4.3%), gate accuracy reported as end-to-end quality (58% vs 98%), in-sample thresholds, and a channel effect on injection false positives that vanished with channel-native content.
No. Two other suspected confounds did not change the conclusions, and the authors report them alongside the confirmed issues, which is part of why the paper is framed as a self-audit rather than only a model comparison.
All cases, raw outputs, and analysis code are available at github.com/David-DL-Space/sys1-eval, the repository named in the paper's comment, so the paired tests and the audit's corrected numbers can be rerun.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: Secret detection with Jev, tested
More Jev resources like this
2GitHub stars
An automated framework that traces adversarial context through Jev-backed program decisions to downstream effects. Across 228 real-world Jev applications and 2,273 test cases, 81.5% show decision flips, 98.6% with potential security impact, and Decision Flip Rate beats the strongest baseline by 50.4 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
xiaoxiaohg8d/JevProbe-Python
xiaoxiaohg8d/JevProbe-PythonA controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An empirical comparison of Jev as a typed judge for retrospective agent-trace security against four generative judges on 5,219 trajectories. Jev averages positive-class F1 77.8 versus 74.1 for GLM-5.2, leading on two of four benchmark collections.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An empirical study of whether Jev's decision-level savings transfer to task-level gains, across five agents in ALFWorld and AndroidWorld. Across 4,500 task executions, decision cost and latency fall in all five agents, but success declines in two mobile agents and per-outcome cost rises in one.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonA benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.