akitaonrails/jev-docs-benchmarks
User repository on GitHub
Study documentation and benchmark for various Jev-like products (commercial and open source)
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev and open decision-model benchmarks
AkitaOnRails' study documentation and benchmark harness for Jev-like decision models: frozen inputs, inspectable expected answers, raw response archives and reproducible scoring compare TypeSafe Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
- Stars
- 6
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- akitaonrails
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- Measures rule application, probability behaviour and decision cost across commercial and open decision models.
- Compares Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
- Frozen inputs, inspectable expected answers, raw response archives and reproducible scoring; schema-valid JSON is not treated as correctness.
- Includes a CUDA-versus-ROCm hardware comparison on an RTX 5090 with paired results and reproduction steps.
- The README states it is not an official vendor benchmark, production accuracy guarantee or proof any model is universally best.
Watch out
README reviewed 2026-10-17; the digest ends inside the 'Start here' section, so the result tables, methodology and limitations sections could not be read. No licence is listed in the captured metadata, and every number in the repository is the author's own measurement rather than an audited vendor result.
Questions about this entry
Short answers drawn from the linked project or page.
A study documentation and benchmark repository by AkitaOnRails that measures decision models on how reliably they apply supplied rules, how their probabilities behave, and what their decisions cost, combining source research with the author's own synthetic experiments.
The README names TypeSafe Jev, Microsoft Decision-1, OpenAI Decisions, hosted and local Clef/Clef-flash, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider, Simple-Jev and GLiNER, evaluated on frozen inputs with reproducible scoring.
Classification, support routing, retrieval and similar decision tasks, per the README's goal statement. It asks where each model works, where it fails, and whether quality, latency and operating cost justify using it.
No. The README says explicitly it is not an official vendor benchmark, not a production accuracy guarantee, and not a demonstration that one model is universally best. It also warns that schema-valid JSON is not evidence that a decision is correct.
Yes — a CUDA-versus-ROCm comparison on an RTX 5090, with a results page of paired runs and a reproduction-and-cleanup guide in benchmark/cuda. That is the first item visible in the digest's 'Start here' section.
AkitaOnRails; 6 stars, created 2026-10-10 and pushed 2026-10-15, in Python. GitHub lists no licence in the digest metadata, and the captured README text does not include one, so reuse rights are unclear.
It does not claim vendor endorsement or universal rankings: it presents frozen inputs, expected answers, raw archives and scoring scripts so others can inspect or rerun the comparisons, and it separates source-based research from the author's own experiments.
More Jev resources like this
A benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
zachlandes/trolley-benchHTML
zachlandes/trolley-benchHTMLA Python experiment that adapts Gemma 4 E4B for System One-style decisions: one call takes a situation and a fixed answer list and returns a probability per answer, with per-dataset adapters and held-out results across four datasets.
Jev Benchmarks & Evaluations#community#github#python
Community
omnaiduu/jevPython
omnaiduu/jevPythonA study of script-dependent answers about Taiwan in commercial decision models: 28 statements asked in Traditional Chinese, Simplified Chinese and English to OpenAI, TypeSafe Jev and Cloudflare models, with data, code and a paper.
Jev Benchmarks & Evaluations#community#github#research
The code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.