Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic

Jev directory

CommunityJev Benchmarks & Evaluations

JudgeJev

An educational lab for DeepEval plus Jev: replay 17 recorded Jev evaluations of fictional support cases, inspect raw requests and responses, adjust release gates and drift playback, or run fresh evaluations with your own key.
See highlightsOpen github.com
Stars
1
Language
Python
License
MIT
Reading time
3 min
Published by
Community
Author
shyamsridhar123
Last verified
Oct 2026
Tagscommunitycatalog-importlogicrwdeepevalevaluationdrift-detection
Works forJev for EvaluationJev for Decision gating
Used inJev for Software & DevToolsJev for Customer Support
Languages
Python
Platforms
Browser
Capabilities
Open sourceRequires API key

Highlights

  • The public site replays 17 real recorded Jev evaluations and makes no model calls; its drift sequence is reconstructed and labels reused samples.
  • It opens with a false shipping guarantee that scored 15.1% beside a correct control at 95.5%, showing why a five-pair candidate fails a release gate.
  • Exercises LLMTestCase fields and native JevEval with Noul, ordinal Score and credited Choice questions, plus weights, thresholds and strict mode.
  • Release gates require complete applicable pairs, a minimum pass rate and a maximum mean drop; monitoring separates quality change from ticket-mix change.
  • The repo's Python app makes fresh provider calls, streams progress and stores a local evidence ledger; Python 3.12 and your own Jev key.

Quickstart

bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
cp .env.example .env
python serve.py

Watch out

Grounded in the truncated README and GitHub metadata reviewed on 2026-10-15; the README was cut off after the local-app startup section, the hosted demo makes no live model calls, the gate and drift rules are application logic rather than a monitoring service, and Python 3.12 is required.

Questions about this entry

Short answers drawn from the linked project or page.

From the Jev guides

Original write-ups on what this entry is for.

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai