User repository on GitHub
Conformance and behaviour test harness for Jev / System One decision models (POST /v1/systemone)
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Systemone Probe
A Python conformance and behaviour test harness for Jev and other System One decision models that speak the POST /v1/systemone protocol. It reports contract, correctness, calibration, determinism, batching, robustness and limits.
- Stars
- 0
- Language
- Python
- License
- MIT
- Reading time
- 2 min
- Published by
- Community
- Author
- mokko
- Last verified
- Oct 2026
Highlights
- Points at an endpoint and reports whether it honours the answer contract, grounds on state and holds probabilities still.
- Checks calibration and documented behaviours: many questions per request, no cross-question contamination and the error surface.
- The sample report shows contract, correctness, calibration, determinism, batching, robustness and limits all passing.
- The README notes much of the contract can be checked without labels: probability sums, arg-max agreement, score and confidence.
- MIT-licensed, Python, 0 stars; created and pushed 2026-10-16.
Watch out
README reviewed 2026-10-17; the captured excerpt stops mid-bullet while explaining how ground truth is computed in Python, so install, CLI flags, endpoint authentication and the full metric definitions are not visible. Metadata shows 0 stars.
Questions about this entry
Short answers drawn from the linked project or page.
A Python conformance and behaviour test harness for System One decision models, including Jev and open Jev-like models that speak the POST /v1/systemone wire protocol.
It reports whether the endpoint honours the answer contract, grounds on the state, keeps probabilities still, is calibrated, and supports many questions per request without cross-question contamination.
The sample line lists contract, correctness, calibration, determinism, batching, robustness and limits, each marked PASS, so the report is organised around those seven checks.
The README says much of the contract can be checked without labels, such as probability sums, arg-max agreement, score equals the weighted sum, and confidence reproduced from the answer's own probabilities.
Accuracy and calibration ground truth can be computed in Python for the included checks, according to the README; the captured excerpt does not say whether any tests require external labelled datasets.
The captured README does not include install, endpoint configuration or command-line examples; it only shows a one-line sample report, so the usage steps cannot be verified from the digest.
The repo belongs to mokko, is MIT-licensed, has 0 stars and 0 forks, and was created and last pushed on 2026-10-16, the day before this entry was written.
More Jev resources like this
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
zachlandes/trolley-benchHTML
zachlandes/trolley-benchHTMLThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonAkitaOnRails' study documentation and benchmark harness for Jev-like decision models: frozen inputs, inspectable expected answers, raw response archives and reproducible scoring compare TypeSafe Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
Jev Benchmarks & Evaluations#community#github#python
A benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonA deterministic gridworld that benchmarks System One decision backends, typed calibrated decisions over a declared option set with one forward pass and no decoding, in a sequential loop, with a scripted oracle scoring agreement without labels.
Jev Benchmarks & Evaluations#community#github#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.