User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Jev Harness Lab
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
- Stars
- 3
- Language
- Python
- Published by
- Community
- Author
- Aitejiu
- Last verified
- Sep 2026
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- PyPI jev-mcp exposes scan_injection, bash_risk, and rank_candidates tools via uvx.
- eval/ holds eleven benchmark scripts, twenty-four reports, and datasets such as a 130-command shell-risk set.
- jev-skill-router skill routes tasks to one installed skill using Jev instead of loading every SKILL.md.
- Pairs with the dev.to harness article; about 22,500 calls and $2.19 total in that write-up.
Quickstart
bash
uvx --from git+https://github.com/Aitejiu/jev-harness-lab jev-mcpWatch out
MIT. Small star count; author-run black-box benchmarks on jev-1.13.0 only.
Questions about this entry
Short answers drawn from the linked project or page.
Mixed. The benchmarks use public datasets and the raw result JSONL is committed, so reports can be regenerated. But the README states that some official baselines are simulated implementations, flagged in the report, and the 130-command shell risk set is hand-built rather than a public corpus.
The documented limits are narrow: choice takes at most 255 options, input is text only, state and questions together share roughly 32k tokens, and English dominates. Nothing is documented about other languages, so non-English input is effectively untested rather than unsupported.
That is the design goal. It uses Jev to pick one skill from your local skill directory and returns only that skill's full instructions, so the main model never carries every SKILL.md. The bundled opencode integration suggests also setting agent.build.tools.skill to false to stop the built-in tool re-listing the directory.
The report keeps its negative results: model-difficulty routing scored 51% on RouterBench and is described as carrying no signal, while trajectory failure attribution reached AUROC 0.56 over 1,403 steps. Everything is pinned to jev-1.13.0, and the README advises re-running the evaluations after a model upgrade.
uvx jev-mcp runs the MCP server from PyPI, or uvx --from git+https://github.com/Aitejiu/jev-harness-lab jev-mcp runs it from source. npx skills add Aitejiu/jev-harness-lab --skill jev-skill-router installs the skill. Both need a TYPESAFE_API_KEY.
scan_injection returns block/review/pass for injected instructions in tool or page output, bash_risk scores shell commands on destructive, secret-touching, exfiltration, and irreversibility axes, and rank_candidates reranks up to 10 candidate fragments by relevance.
Run the eval/run_*.py scripts with uv and Python 3.12; results are cached in eval/results/*.jsonl and committed so --report-only rebuilds reports without new calls, while raw datasets download separately per eval/README.md.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: jev-review: a local-first score loop for coding agents
Reddit: One command to wire Jev into Claude Code, Claude Desktop, and Codex.
More Jev resources like this
21GitHub stars
MIT-licensed Python tool router for coding agents that keeps obvious calls on a fast path, asks Jev (default typesafe/jev-1.13 via OpenRouter) for genuinely ambiguous choices, and can run bounded MCTS when multi-step consequences matter. Ships a CLI, MCP server and hook integrations.
Jev Tools & CLIs#community#routing#mcp
1GitHub stars
Python CLI that runs declarative YAML, JSON or JSONL test suites natively on Jev Noul, Choice and Score decisions for RAG faithfulness, hallucination detection and agent safety, with watch mode, parallel batching, a pytest plugin and GitHub Actions PR reporting.
Jev Benchmarks & Evaluations#community#cli#ci-cd
Community
00200200/jev-judgePython
00200200/jev-judgePython0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript22GitHub stars
An MCP proxy and hooks adapter that screens an agent's tool calls before they run and the tool results before the agent reads them, with thresholds in a policy file.
Jev Agents & MCP#community#mcp#security
15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPythonFrom the Jev guides
Original write-ups on what this entry is for.
The fastest-growing corner of the Jev ecosystem is coding agents. A tour of the MCP servers, tool-call gates, context pruners, and skill rankers built for that loop.
Jev Tools & CLIs#agents#mcp#coding-agent
Read article
From the Jev community
Posts from builders shipping with Jev right now.