User repository on GitHub
Behavioral contracts for TypeSafe Jev — pin production expectations, eval model upgrades, catch flips and confidence regressions.
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jevcheck
pytest-style behavioral contracts for Jev: pin a production contract of baseline model plus fixtures, eval a candidate model against it, and fail the upgrade when answers flip or confidence regresses.
- Stars
- 2
- Language
- Python
- License
- MIT
- Reading time
- 2 min
- Published by
- Community
- Author
- sathariels
- Last verified
- Oct 2026
Highlights
- Reports unchanged, confidence regressions or answer flips with exact diffs such as billing→general and 0.94→0.71, and exits nonzero on failure.
- Rejects floating aliases jev-latest and jev-preview unless --allow-unpinned is passed; concrete pins require exact model identity.
- v0.2 adds two-model execution: record snaps a baseline model's answers, then compare evals a candidate against that snapshot or fetches both live.
- The repo's jev-1.13 and jev-1.14 strings are unverified example pin labels in fixtures; a documented pin example is jev-1.13.0.
- eval --answers replays recorded fixtures without a key for CI, and auth for live calls is TYPESAFE_API_KEY only.
Quickstart
bash
pip install jevcheck
jevcheck eval fixtures/support-triage.json --candidate-model jev-1.14Watch out
Copy reviewed against the truncated repo README on 2026-10-15; the excerpt cuts off at the end of the record section. The jev-1.13 and jev-1.14 labels used by fixtures are explicitly unverified examples, and the HOL Guard badge is an external registry widget.
Questions about this entry
Short answers drawn from the linked project or page.
Probabilities and model versions move, so a raw 0.94 is not a release decision. jevcheck records a production contract — baseline model, fixtures and expected answers — and evals a candidate against that fixture before you upgrade.
pip install jevcheck, write a contract against the model you ship, then run jevcheck eval fixtures/support-triage.json --candidate-model jev-1.14. Exit 0 means compatible, so you can change the pin; exit 1 means read the diffs and do not upgrade.
TYPESAFE_API_KEY only for live Jev calls, and the README says never to commit it. Unit tests mock the network, and eval --answers replays recorded answers so CI can run without a key.
Floating aliases undermine the contract, so they are rejected unless you pass --allow-unpinned. Concrete pins require the response's model to match exactly; with the opt-in, the report prints the resolved model ID.
Two-model execution: jevcheck record writes a baseline model's answers to a replay JSON, then compare evals a candidate against that snapshot, or fetches both models live. The v0.1 eval path is unchanged.
Each report is unchanged, confidence regression or answer flip, with exact diffs like billing→general and 0.94→0.71 and a nonzero exit on failure — the README describes it as pytest for Jev.
Not verified. They are example pin labels used by the repo's replay fixtures; a documented pin from the 2026-09-19 model list is jev-1.13.0, and the README warns the fixture labels match replay data only.
More Jev resources like this
A reproducible lab measuring how several coding agents coordinate on one repository, plus Médula, a kernel that asks TypeSafe Jev before every write whether a change collides with what the other agents are doing.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
joaquinruiz/medulaPython
joaquinruiz/medulaPythonBenchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonA Python research harness evaluating Jev (TypeSafe System One, jev-1.13.0) as a relevance judge for task-level tool spaces. One API call scores every tool with a yes/no noul question per tool, and the repo benchmarks keep-all versus per-request, per-task and trace-based spaces on TGB and BFCL.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
xuan7zhang/jev-toolspacePython
xuan7zhang/jev-toolspacePythonLatitude is an open-source observability platform for AI agents: one line of telemetry captures traces, failing traces group into tracked signals, and fixes are dispatched to coding agents and replayed against real traces.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
latitude-dev/latitude-llmTypeScript
latitude-dev/latitude-llmTypeScriptA Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.