Jev directory
CommunityJev Benchmarks & Evaluations
LLM Output Evaluation with Clef
Pytest-based framework that judges mocked LLM outputs with Cloudflare clef-flash, a 9B decision model served locally by Ollama. Metrics turn typed probabilities into values between 0 and 1 and PASS/FAIL against a threshold.
- Stars
- 0
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Runs clef-flash (~9.7 GB) locally in Ollama through Docker, with no hosted judge API required.
- Metric kinds noul, score and choice convert typed answers into value in [0, 1] and PASS/FAIL.
- Ships rubrics for RAG correctness, safety, style, classification, routing, tool choice and A/B.
- Judge answers are cached per model digest: about 20 minutes on the first CPU run, seconds after.
- Requires Python 3.11+, uv, Docker with compose, and roughly 10 GB of disk for the model.
Questions about this entry
Short answers drawn from the linked project or page.
Run make install for dependencies, make env to create .env from .env.example, make pull to start Ollama in Docker and download clef-flash, then make test-unit for framework tests or make test-evals for evals with the model. Results land in reports/evals.html.
Python 3.11+, uv, Docker with docker compose, and about 10 GB of disk for the model. CLEF_TIMEOUT defaults to 600 seconds per request because CPU inference is slow.
About 20 minutes on CPU for the first run. The model is deterministic, so answers are cached in .clef_cache per model digest and later runs take seconds. Set CLEF_CACHE to control reuse.
RAG and correctness metrics such as faithfulness, answer_relevance, context_relevance, correctness, completeness and summary_coverage; safety metrics such as toxicity, pii_leak, prompt_injection, harmful_compliance and refusal; style metrics; and decision metrics for classification, routing, escalation, tool_choice and tool_arguments.
Yes. The docs/custom_metrics.md guide covers adding a new metric in code or in YAML and picking a threshold. Case suites are YAML files under tests/data, one per category, loaded by load_suite().
Yes. Setting CLEF_LOG_STATE=false stops case data from appearing in logs and reports. Each run otherwise writes a log file at DEBUG level and JSON lines of test results through JSONLogger.
More Jev resources like this
Independent experiment scoring 400 real job postings against 30 fictional résumés with Jev, producing 12,000 evaluations in about 197 seconds for roughly $0.17, with a 100-posting answer key and an analysis of what the model got wrong.
Jev Benchmarks & Evaluations#github#madewithjev#evaluation
Communitykaucap08/jev-job-matching
Independent audit repository testing whether Jev's probabilities are calibrated on EIA weekly petroleum reports, comparing Jev via LangChain TypeSafeClassifier against an LLM-as-classifier, rules, and climatology baselines.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Communityn9omi/jev-calibration-audit
Pkl-based divergence tracker for a small TypeSafe AI API experiment. Four messages are scored by the is_urgent question on jev-latest and compared with a pinned baseline, using sanity bands and tolerances to flag behaviour drift.
Jev Benchmarks & Evaluations#github#evaluation#calibration
Communitynull-hype/typesafe-experiment
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
A timestamped benchmark and trading arena that replays resolved prediction markets: models commit to probabilities using only information available at the time, and realized P&L is the headline metric with Brier, log-loss and ECE as checks.
Jev Benchmarks & Evaluations#community#github#python
Community
ruyianry/JevGymPython
ruyianry/JevGymPythonGTM workflow experiments with Jev from deepdave98: weekly datasets, prompts, and benchmark results that test Jev against Claude on lead triage and routing, deal risk, reply triage, MEDDPICC from call summaries, duplicate accounts, and closed-lost reactivation.
Jev Benchmarks & Evaluations#community#github#gtm
Community
deepdave98/jev-playgroundPython
deepdave98/jev-playgroundPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.