User repository on GitHub
Vitest for LLM outputs. Sub-100ms, deterministic CI/CD evaluations powered by TypeSafe Jev.
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Jev Judge — 00200200
Python CLI that runs declarative YAML, JSON or JSONL test suites natively on Jev Noul, Choice and Score decisions for RAG faithfulness, hallucination detection and agent safety, with watch mode, parallel batching, a pytest plugin and GitHub Actions PR reporting.
- Stars
- 1
- Language
- Python
- License
- MIT
- Published by
- Community
- Last verified
- Oct 2026
About this project
Vitest for LLM outputs. Sub-100ms, deterministic CI/CD evaluations powered by TypeSafe Jev (System One). Running automated evaluations (evaluating RAG pipelines, agents, or chatbot outputs) in CI/CD with generative models like GPT-4o or Claude 3.5 Sonnet is broken: ⏱️ Too Slow: 500 test cases take 12 to 18 minutes to run in GitHub Actions. 💸 Too Expensive: Costs $20 to $50 per PR, making it impossible to run on every commit.
Quoted from the project's README. Read the full README.
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- Built-in evaluators cover faithfulness, hallucination, relevance, safety, toxicity, PII/secret leaks and semantic drift.
- README claims sub-100ms decisions at about $0.00004 each, and suites run entirely offline in a mock mode.
- Ships an MCP server with jev_evaluate and jev_run_suite tools plus a LangChain callback and a LlamaIndex evaluator.
- MIT licensed, Python 3.10+; CLI exports JUnit, JSON, GitHub annotations and self-contained HTML reports.
Watch out
MIT licensed. Live inference needs a TypeSafe API key, and the latency and cost comparison table is the project's own measured claim.
Questions about this entry
Short answers drawn from the linked project or page.
Yes. jev-judge includes an offline mock mode, so you can scaffold suites with jev-judge init and run them before connecting a TypeSafe key for live inference.
The CLI emits JUnit, JSON and GitHub annotation formats, ships a GitHub Action that comments pass/fail tables on pull requests, and works as a pre-commit hook or a pytest plugin.
pip install jev-judge, or uvx jev-judge test evals/ with no install. jev-judge init writes evals/rag_evals.yaml, evals/agent_evals.yaml and .github/workflows/evals.yml. A suite has a name, an optional threshold, and tests with context, input, output and assertions.
They are the project's own measurement over 1,000 paired RAG faithfulness and hallucination items: Jev at 48 ms p50 and $0.04 per 1k evaluations, GPT-4o-mini at 820 ms and $1.50, Claude 3.5 Sonnet at 1,420 ms and $18.00, and GPT-4o at 1,850 ms and $25.00. The README's positioning claim is that 10,000 evals cost about 40 cents.
Built-in evaluators cover faithfulness or groundedness, hallucination, relevance, safety and prompt injection, toxicity, PII and credential or secret leaks, and semantic drift, plus any natural-language custom assertion. In YAML, faithfulness: pass and hallucination: false are Noul checks and relevance: 5 is a Score check on a 1 to 5 rubric.
Yes. It ships an MCP server exposing jev_evaluate for sub-100ms checks of faithfulness, hallucination and safety, and jev_run_suite to run test suites on demand. Add it with claude mcp add --scope user, or as an mcpServers entry in Claude Desktop, Cursor or Windsurf running uvx jev-judge mcp.
Instantiate Judge and call evaluate with context, input, output and an assertions dict, then read passed, duration_ms and per-decision probabilities. There is a jev_judge pytest fixture, a JevLangChainEvaluator for LCEL chains, a LlamaIndex evaluator, and self-contained HTML reports via --output-html or --format html with zero external requests.
Other listings of this project
Forks and copies of this project in the directory. This entry is the canonical one.
0GitHub stars
Dependency-free Python 3 CLI that hands a JSON state plus 1 to 16 named Noul, Choice or Score questions to Jev and prints typed answers, usage and elapsed time without echoing the state back.
Jev Agents & MCP#community#awesomejev#catalog-import
Community
ohmyjiro/jev-judgePython
ohmyjiro/jev-judgePythonMore Jev resources like this
3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPython15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython0GitHub stars
Reproducible benchmark asking which approved agent answers changed meaning after a config edit: one Jev Choice (same, fact_differs, action_differs, specificity_differs) caught all 19 real changes with 13 false alarms across 109 before/after pairs, versus 33 for a three-tier stack and 19 for an LLM judge.
Jev Benchmarks & Evaluations#community#benchmark#regression-testing
24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript8.3kGitHub stars
Agentic search evaluation inside Zilliz DeepSearcher: a standalone harness replaces the ChainOfRAG reflection call with one Jev Noul per round to decide whether to stop within a seven-round budget, then compares stopping behavior, evidence recall and decision cost on 100 sampled 2WikiMultiHopQA queries.
Jev Benchmarks & Evaluations#community#rag#evaluation
Community
zilliztech/deep-searcherPython
zilliztech/deep-searcherPythonFrom the Jev community
Posts from builders shipping with Jev right now.