User repository on GitHub
- Forks
- 11
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev-as-a-Judge with Comet Opik
A worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
- Stars
- 67
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- patchy631
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceRequires API key
Highlights
- Ten frozen synthetic traces cover correct answers, invented policies, false action claims, ambiguous wording, and an evaluator-injection attempt.
- Deterministic checks run before paying for semantic evaluation; one Jev request evaluates three yes/no criteria plus an ordered helpfulness rubric.
- The offline demo needs no account or network: python -m jev_judge.cli --mode demo; its probabilities are hand-authored and labeled DEMO-hand-authored-not-Jev.
- Live mode makes nine initial judge requests and records raw answers, the resolved model, and a rubric version; failures become judge_error, not scores.
- Maps values to separate Opik metrics through a custom BaseMetric; tested with Opik 2.2.71 and Python 3.12 against a fake Jev transport.
Quickstart
bash
export TYPESAFE_API_KEY="your-key"
export JEV_MODEL="jev-latest"
python -m jev_judge.cli --mode live --output results/live.jsonlWatch out
README reviewed 2026-10-15; the author describes this as an evaluator tutorial, not a production guardrail, and it does not issue refunds. No license is imposed, the 0.2/0.8 verdict boundaries and 0.6 confidence threshold are illustrative and uncalibrated, and authenticated Jev calls plus real Opik uploads were not run when building it.
Questions about this entry
Short answers drawn from the linked project or page.
A small, auditable example of evaluating refund-support traces with Jev and recording the results as a Comet Opik experiment. Ten synthetic frozen traces include correct answers, invented policies, false actions, and an injection attempt.
Three yes/no criteria, grounded, addresses_request, and action_honest, plus one ordered helpfulness rubric, all evaluated in a single Jev request, with each value mapped to a separate Opik metric through a custom BaseMetric.
Yes. python -m jev_judge.cli --mode demo needs no dependencies or network, and the unittest suite runs offline; the demo probabilities are hand-authored and explicitly labeled DEMO-hand-authored-not-Jev, so they test plumbing, not the model.
It sends the synthetic request, policy, tool results, and final answer to TypeSafe and may incur charges: nine initial judge requests, plus retries if any. Failures are marked judge_error, routed to review, and produce a nonzero exit code.
A noul value of 0.98 is the model's probability of yes to the whole proposition, not the share of grounded claims; score confidence is not the probability that the judge is correct, and the thresholds are illustrative and uncalibrated.
Pass --data path/to/cases.jsonl; each line needs id, request, policy, tool_calls, and final_answer. Redact personal data first, and note that expected_verdict fields are tutorial annotations, not human labels unless you collect them.
No: the README calls it an evaluator tutorial that does not issue refunds, with no automated escalation or background worker implemented, no license imposed, and CI that runs offline tests only.
More Jev resources like this
An educational lab for DeepEval plus Jev: replay 17 recorded Jev evaluations of fictional support cases, inspect raw requests and responses, adjust release gates and drift playback, or run fresh evaluations with your own key.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
shyamsridhar123/judgejevPython
shyamsridhar123/judgejevPythonA Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonA pre-registered independent evaluation of Jev on intent classification and confidence-gated cascades against nano-class and frontier LLMs and a supervised encoder, plus a second study of confidence under prompt injection.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
ickma2311/jev-baselines-evalPython
ickma2311/jev-baselines-evalPythonpytest-style behavioral contracts for Jev: pin a production contract of baseline model plus fixtures, eval a candidate model against it, and fail the upgrade when answers flip or confidence regresses.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
sathariels/jevcheckPython
sathariels/jevcheckPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.